[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:41:34 dist-test-slave-k6lg krb5kdc[6029](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:41:34 dist-test-slave-k6lg krb5kdc[6029](info): set up 2 sockets
Jun 20 18:41:34 dist-test-slave-k6lg krb5kdc[6029](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 18:41:35.591969 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:41:35 dist-test-slave-k6lg krb5kdc[6029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:41:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:41:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
I20260620 18:41:35.761869 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:38331
--webserver_interface=127.5.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:41265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:38331
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:36.087355 6042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:36.087646 6042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:36.087792 6042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:36.095122 6042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:41:36.095227 6042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:36.095324 6042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:36.095408 6042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:41:36.095491 6042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:41:36.105515 6042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:41265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:38331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:38331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:41:36.106977 6042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:36.108798 6042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:36.121872 6047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:36.121961 6048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:36.122435 6050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:36.122720 6042 server_base.cc:1034] running on GCE node
I20260620 18:41:36.123577 6042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:36.125001 6042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:36.126266 6042 hybrid_clock.cc:648] HybridClock initialized: now 1781980896126187 us; error 53 us; skew 500 ppm
I20260620 18:41:36.126616 6042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:36 dist-test-slave-k6lg krb5kdc[6029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980896, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:36.129721 6042 init.cc:377] Logged in from keytab as kudu/127.5.225.190@KRBTEST.COM (short username kudu)
I20260620 18:41:36.131248 6042 webserver.cc:460] Webserver started at http://127.5.225.190:45937/ using document root <none> and password file <none>
I20260620 18:41:36.131956 6042 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:36.132088 6042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:36.132436 6042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:36.135454 6042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "12d8438bb77b465aa0f3fbcc12b420a2"
format_stamp: "Formatted at 2026-06-20 18:41:36 on dist-test-slave-k6lg"
server_key: "426fb59f9aa787a4acc9ef06b7004264"
server_key_iv: "d76f8ca182e9aff16388cfaeb471c0e6"
server_key_version: "encryptionkey@0"
I20260620 18:41:36.136219 6042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "12d8438bb77b465aa0f3fbcc12b420a2"
format_stamp: "Formatted at 2026-06-20 18:41:36 on dist-test-slave-k6lg"
server_key: "426fb59f9aa787a4acc9ef06b7004264"
server_key_iv: "d76f8ca182e9aff16388cfaeb471c0e6"
server_key_version: "encryptionkey@0"
I20260620 18:41:36.141479 6042 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:41:36.144907 6057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:36.146214 6042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 18:41:36.146387 6042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "12d8438bb77b465aa0f3fbcc12b420a2"
format_stamp: "Formatted at 2026-06-20 18:41:36 on dist-test-slave-k6lg"
server_key: "426fb59f9aa787a4acc9ef06b7004264"
server_key_iv: "d76f8ca182e9aff16388cfaeb471c0e6"
server_key_version: "encryptionkey@0"
I20260620 18:41:36.146574 6042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:36.163997 6042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:36.168216 6042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:36.168511 6042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:36.187680 6042 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:38331
I20260620 18:41:36.187703 6108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:38331 every 8 connection(s)
I20260620 18:41:36.189142 6042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:41:36.193095 6109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:36.198530 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 6042
I20260620 18:41:36.198845 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:41:36.201356 6022 external_mini_cluster.cc:1428] Setting key 68459fb5b08dad8e86e3c52c9d2a684e
I20260620 18:41:36.205440 6109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12d8438bb77b465aa0f3fbcc12b420a2: Bootstrap starting.
I20260620 18:41:36.208877 6109 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12d8438bb77b465aa0f3fbcc12b420a2: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:36.209915 6109 log.cc:824] T 00000000000000000000000000000000 P 12d8438bb77b465aa0f3fbcc12b420a2: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:36.212710 6109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12d8438bb77b465aa0f3fbcc12b420a2: No bootstrap required, opened a new log
I20260620 18:41:36.222244 6109 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12d8438bb77b465aa0f3fbcc12b420a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12d8438bb77b465aa0f3fbcc12b420a2" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 38331 } }
I20260620 18:41:36.222762 6109 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12d8438bb77b465aa0f3fbcc12b420a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:36.222867 6109 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12d8438bb77b465aa0f3fbcc12b420a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12d8438bb77b465aa0f3fbcc12b420a2, State: Initialized, Role: FOLLOWER
I20260620 18:41:36.223511 6109 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12d8438bb77b465aa0f3fbcc12b420a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12d8438bb77b465aa0f3fbcc12b420a2" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 38331 } }
I20260620 18:41:36.223821 6109 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12d8438bb77b465aa0f3fbcc12b420a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:41:36.223977 6109 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12d8438bb77b465aa0f3fbcc12b420a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:41:36.224159 6109 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12d8438bb77b465aa0f3fbcc12b420a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:36.226867 6109 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12d8438bb77b465aa0f3fbcc12b420a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12d8438bb77b465aa0f3fbcc12b420a2" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 38331 } }
I20260620 18:41:36.227458 6109 leader_election.cc:302] T 00000000000000000000000000000000 P 12d8438bb77b465aa0f3fbcc12b420a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12d8438bb77b465aa0f3fbcc12b420a2; no voters:
I20260620 18:41:36.228057 6109 leader_election.cc:288] T 00000000000000000000000000000000 P 12d8438bb77b465aa0f3fbcc12b420a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:41:36.228227 6114 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12d8438bb77b465aa0f3fbcc12b420a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:41:36.229485 6114 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12d8438bb77b465aa0f3fbcc12b420a2 [term 1 LEADER]: Becoming Leader. State: Replica: 12d8438bb77b465aa0f3fbcc12b420a2, State: Running, Role: LEADER
I20260620 18:41:36.230098 6114 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12d8438bb77b465aa0f3fbcc12b420a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12d8438bb77b465aa0f3fbcc12b420a2" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 38331 } }
I20260620 18:41:36.230456 6109 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12d8438bb77b465aa0f3fbcc12b420a2 [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 18:41:36 dist-test-slave-k6lg krb5kdc[6029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:41:36.233969 6116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12d8438bb77b465aa0f3fbcc12b420a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12d8438bb77b465aa0f3fbcc12b420a2. Latest consensus state: current_term: 1 leader_uuid: "12d8438bb77b465aa0f3fbcc12b420a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12d8438bb77b465aa0f3fbcc12b420a2" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 38331 } } }
I20260620 18:41:36.234084 6115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12d8438bb77b465aa0f3fbcc12b420a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12d8438bb77b465aa0f3fbcc12b420a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12d8438bb77b465aa0f3fbcc12b420a2" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 38331 } } }
I20260620 18:41:36.234403 6116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12d8438bb77b465aa0f3fbcc12b420a2 [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:36.234485 6115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12d8438bb77b465aa0f3fbcc12b420a2 [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:36.235488 6121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:41:36.248808 6111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:36.206137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54298 (local address 127.5.225.190:38331)
0620 18:41:36.207173 (+ 1036us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:36.207191 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:36.208469 (+ 1278us) server_negotiation.cc:407] Connection header received
0620 18:41:36.217637 (+ 9168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:36.217665 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:36.218243 (+ 578us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:36.218720 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:36.221621 (+ 2901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:36.223625 (+ 2004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:36.225497 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:36.226566 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:36.226768 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:36.234860 (+ 8092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:36.234876 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:36.234908 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:36.235013 (+ 105us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:36.239741 (+ 4728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:36.241116 (+ 1375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:36.241129 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:36.241140 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:36.241405 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:36.242095 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:36.242103 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:36.242109 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:36.243019 (+ 910us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:36.243354 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:36.246512 (+ 3158us) server_negotiation.cc:299] Negotiation successful
0620 18:41:36.247072 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":446,"threads_started":1}
I20260620 18:41:36.252849 6121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:41:36.261931 6121 catalog_manager.cc:1347] Generated new cluster ID: a6d70f39e0ad4b8fba0b1135daca9527
I20260620 18:41:36.262068 6121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:41:36.284747 6121 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:41:36.286614 6121 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:41:36.293955 6121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12d8438bb77b465aa0f3fbcc12b420a2: Generated new TSK 0
I20260620 18:41:36.295061 6121 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab.
Entry for principal kudu/127.5.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab.
I20260620 18:41:36.341076 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.129:0
--local_ip_for_outbound_sockets=127.5.225.129
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:38331
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:41265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:36.663017 6135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:36.663285 6135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:36.663381 6135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:36.670686 6135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:36.670794 6135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:36.670908 6135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.129
I20260620 18:41:36.680568 6135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:41265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:38331
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.129
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:41:36.681742 6135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:36.683008 6135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:36.695189 6143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:36.695215 6141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:36.695392 6140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:36.696285 6135 server_base.cc:1034] running on GCE node
I20260620 18:41:36.696900 6135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:36.697775 6135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:36.698975 6135 hybrid_clock.cc:648] HybridClock initialized: now 1781980896698944 us; error 30 us; skew 500 ppm
I20260620 18:41:36.699275 6135 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:36 dist-test-slave-k6lg krb5kdc[6029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980896, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:36.702334 6135 init.cc:377] Logged in from keytab as kudu/127.5.225.129@KRBTEST.COM (short username kudu)
I20260620 18:41:36.703727 6135 webserver.cc:460] Webserver started at http://127.5.225.129:33039/ using document root <none> and password file <none>
I20260620 18:41:36.704385 6135 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:36.704530 6135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:36.704838 6135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:36.707732 6135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-0/data/instance:
uuid: "316d864121934ca08c9dcb1e685ec95f"
format_stamp: "Formatted at 2026-06-20 18:41:36 on dist-test-slave-k6lg"
server_key: "8e25ef4bb3c540e62f3de3ed9e8328e5"
server_key_iv: "1ad17ee70ed65374f4daf03e1c3c31c2"
server_key_version: "encryptionkey@0"
I20260620 18:41:36.708560 6135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance:
uuid: "316d864121934ca08c9dcb1e685ec95f"
format_stamp: "Formatted at 2026-06-20 18:41:36 on dist-test-slave-k6lg"
server_key: "8e25ef4bb3c540e62f3de3ed9e8328e5"
server_key_iv: "1ad17ee70ed65374f4daf03e1c3c31c2"
server_key_version: "encryptionkey@0"
I20260620 18:41:36.713685 6135 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:41:36.717343 6150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:36.719017 6135 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 18:41:36.719202 6135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-0/wal
uuid: "316d864121934ca08c9dcb1e685ec95f"
format_stamp: "Formatted at 2026-06-20 18:41:36 on dist-test-slave-k6lg"
server_key: "8e25ef4bb3c540e62f3de3ed9e8328e5"
server_key_iv: "1ad17ee70ed65374f4daf03e1c3c31c2"
server_key_version: "encryptionkey@0"
I20260620 18:41:36.719430 6135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:36.747737 6135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:36.752431 6135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:36.752732 6135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:36.753897 6135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:41:36.755443 6135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:36.755546 6135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:36.755692 6135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:36.755766 6135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:36.789178 6135 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.129:44293
I20260620 18:41:36.789232 6262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.129:44293 every 8 connection(s)
I20260620 18:41:36.790575 6135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
I20260620 18:41:36.797526 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 6135
I20260620 18:41:36.797750 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance
I20260620 18:41:36.798560 6022 external_mini_cluster.cc:1428] Setting key a40fc56199ef6acc0517c9c7b4a902cf
Jun 20 18:41:36 dist-test-slave-k6lg krb5kdc[6029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980896, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:41:36.812615 6265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:36.793155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:45285 (local address 127.5.225.190:38331)
0620 18:41:36.793771 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:36.793775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:36.794540 (+ 765us) server_negotiation.cc:407] Connection header received
0620 18:41:36.796340 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:36.796346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:36.796457 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:36.796605 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:36.798565 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:36.799457 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:36.801454 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:36.801850 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:36.801953 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:36.806122 (+ 4169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:36.806127 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:36.806130 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:36.806204 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:36.808978 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:36.810008 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:36.810012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:36.810015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:36.810147 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:36.810758 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:36.810764 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:36.810768 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:36.811084 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:36.811198 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:36.812188 (+ 990us) server_negotiation.cc:299] Negotiation successful
0620 18:41:36.812358 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":226,"threads_started":1}
I20260620 18:41:36.814308 6266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:36.793668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:38331 (local address 127.5.225.129:45285)
0620 18:41:36.794391 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:36.794434 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:36.796088 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:36.796800 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:36.796808 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:36.797485 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:36.798354 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:36.798393 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:36.799599 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:36.799604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:36.801302 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:36.801321 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:36.802076 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:36.802085 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:36.802451 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:36.803218 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:36.803252 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:36.805866 (+ 2614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:36.809130 (+ 3264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:36.809142 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:36.809162 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:36.809847 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:36.810329 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:36.810337 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:36.810343 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:36.810595 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:36.811220 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:36.811230 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:36.811717 (+ 487us) client_negotiation.cc:769] Sending connection context
0620 18:41:36.812070 (+ 353us) client_negotiation.cc:240] Negotiation successful
0620 18:41:36.812695 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":279,"threads_started":1}
WARNING: no policy specified for kudu/127.5.225.130@KRBTEST.COM; defaulting to no policy
I20260620 18:41:36.815927 6263 heartbeater.cc:347] Connected to a master server at 127.5.225.190:38331
I20260620 18:41:36.816193 6263 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:36.816990 6263 heartbeater.cc:511] Master 127.5.225.190:38331 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.130@KRBTEST.COM" created.
I20260620 18:41:36.819350 6074 ts_manager.cc:194] Registered new tserver with Master: 316d864121934ca08c9dcb1e685ec95f (127.5.225.129:44293)
I20260620 18:41:36.821530 6074 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.129@KRBTEST.COM'} at 127.5.225.129:45285
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab.
Entry for principal kudu/127.5.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab.
I20260620 18:41:36.841006 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.130:0
--local_ip_for_outbound_sockets=127.5.225.130
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:38331
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:41265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:37.161242 6269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:37.161582 6269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:37.161763 6269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:37.169101 6269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:37.169255 6269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:37.169448 6269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.130
I20260620 18:41:37.179323 6269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:41265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:38331
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.130
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:41:37.180662 6269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:37.182076 6269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:37.195958 6277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:37.196204 6275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:37.196569 6274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:37.196789 6269 server_base.cc:1034] running on GCE node
I20260620 18:41:37.197495 6269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:37.198589 6269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:37.199823 6269 hybrid_clock.cc:648] HybridClock initialized: now 1781980897199754 us; error 61 us; skew 500 ppm
I20260620 18:41:37.200176 6269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:37 dist-test-slave-k6lg krb5kdc[6029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980897, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:37.203955 6269 init.cc:377] Logged in from keytab as kudu/127.5.225.130@KRBTEST.COM (short username kudu)
I20260620 18:41:37.207278 6269 webserver.cc:460] Webserver started at http://127.5.225.130:34275/ using document root <none> and password file <none>
I20260620 18:41:37.207995 6269 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:37.208144 6269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:37.208456 6269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:37.211531 6269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-1/data/instance:
uuid: "c0a012bdc3004cedb75e91cf4d044f3a"
format_stamp: "Formatted at 2026-06-20 18:41:37 on dist-test-slave-k6lg"
server_key: "210b83d5a1408c0959b2ca5f208de935"
server_key_iv: "b84d353e8e2b19d54095002208d6eb57"
server_key_version: "encryptionkey@0"
I20260620 18:41:37.212285 6269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance:
uuid: "c0a012bdc3004cedb75e91cf4d044f3a"
format_stamp: "Formatted at 2026-06-20 18:41:37 on dist-test-slave-k6lg"
server_key: "210b83d5a1408c0959b2ca5f208de935"
server_key_iv: "b84d353e8e2b19d54095002208d6eb57"
server_key_version: "encryptionkey@0"
I20260620 18:41:37.217796 6269 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:41:37.221578 6284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:37.223256 6269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 18:41:37.223428 6269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-1/wal
uuid: "c0a012bdc3004cedb75e91cf4d044f3a"
format_stamp: "Formatted at 2026-06-20 18:41:37 on dist-test-slave-k6lg"
server_key: "210b83d5a1408c0959b2ca5f208de935"
server_key_iv: "b84d353e8e2b19d54095002208d6eb57"
server_key_version: "encryptionkey@0"
I20260620 18:41:37.223654 6269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:37.269457 6269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:37.273787 6269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:37.274109 6269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:37.275177 6269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:41:37.276620 6269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:37.276726 6269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:37.276870 6269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:37.276945 6269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:37.310698 6269 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.130:38703
I20260620 18:41:37.310779 6396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.130:38703 every 8 connection(s)
I20260620 18:41:37.311952 6269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
I20260620 18:41:37.318284 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 6269
I20260620 18:41:37.318478 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance
I20260620 18:41:37.319015 6022 external_mini_cluster.cc:1428] Setting key 0b21a9ff8b6aa6237398e0750aa7c31f
Jun 20 18:41:37 dist-test-slave-k6lg krb5kdc[6029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980897, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:41:37.332777 6399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:37.314518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:52907 (local address 127.5.225.190:38331)
0620 18:41:37.314972 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:37.314976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:37.316038 (+ 1062us) server_negotiation.cc:407] Connection header received
0620 18:41:37.317938 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:37.317942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:37.318010 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:37.318111 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:37.320009 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:37.320856 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:37.322490 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:37.322950 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:37.323092 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:37.326892 (+ 3800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:37.326897 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:37.326901 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:37.326938 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:37.329685 (+ 2747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:37.330530 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:37.330536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:37.330539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:37.330666 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:37.331096 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:37.331102 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:37.331110 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:37.331398 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:37.331562 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:37.332192 (+ 630us) server_negotiation.cc:299] Negotiation successful
0620 18:41:37.332401 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":223,"threads_started":1}
I20260620 18:41:37.333770 6400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:37.315104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:38331 (local address 127.5.225.130:52907)
0620 18:41:37.315869 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:37.315923 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:37.317638 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:37.318432 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:37.318445 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:37.319166 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:37.319860 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:37.319876 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:37.321012 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:37.321018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:37.322302 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:37.322314 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:37.323173 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:37.323182 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:37.323421 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:37.324200 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:37.324220 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:37.326669 (+ 2449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:37.329833 (+ 3164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:37.329842 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:37.329858 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:37.330394 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:37.330770 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:37.330776 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:37.330779 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:37.330963 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:37.331517 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:37.331525 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:37.331852 (+ 327us) client_negotiation.cc:769] Sending connection context
0620 18:41:37.332104 (+ 252us) client_negotiation.cc:240] Negotiation successful
0620 18:41:37.332635 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":355,"threads_started":1}
WARNING: no policy specified for kudu/127.5.225.131@KRBTEST.COM; defaulting to no policy
I20260620 18:41:37.335430 6397 heartbeater.cc:347] Connected to a master server at 127.5.225.190:38331
I20260620 18:41:37.335801 6397 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:37.336699 6397 heartbeater.cc:511] Master 127.5.225.190:38331 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.131@KRBTEST.COM" created.
I20260620 18:41:37.338131 6074 ts_manager.cc:194] Registered new tserver with Master: c0a012bdc3004cedb75e91cf4d044f3a (127.5.225.130:38703)
I20260620 18:41:37.339010 6074 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.130@KRBTEST.COM'} at 127.5.225.130:52907
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab.
Entry for principal kudu/127.5.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab.
I20260620 18:41:37.357098 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.131:0
--local_ip_for_outbound_sockets=127.5.225.131
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:38331
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:41265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:37.681690 6403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:37.681946 6403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:37.682106 6403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:37.689083 6403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:37.689211 6403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:37.689429 6403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.131
I20260620 18:41:37.699096 6403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:41265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:38331
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.131
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:41:37.700367 6403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:37.701759 6403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:37.714882 6408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:37.715101 6411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:37.714960 6409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:37.716171 6403 server_base.cc:1034] running on GCE node
I20260620 18:41:37.716795 6403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:37.717715 6403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:37.718938 6403 hybrid_clock.cc:648] HybridClock initialized: now 1781980897718877 us; error 53 us; skew 500 ppm
I20260620 18:41:37.719336 6403 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:37 dist-test-slave-k6lg krb5kdc[6029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980897, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:37.722612 6403 init.cc:377] Logged in from keytab as kudu/127.5.225.131@KRBTEST.COM (short username kudu)
I20260620 18:41:37.724109 6403 webserver.cc:460] Webserver started at http://127.5.225.131:39715/ using document root <none> and password file <none>
I20260620 18:41:37.724792 6403 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:37.724892 6403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:37.725155 6403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:37.728086 6403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-2/data/instance:
uuid: "49c4037cd9c6478483d52323e0885eb8"
format_stamp: "Formatted at 2026-06-20 18:41:37 on dist-test-slave-k6lg"
server_key: "29aeb233a6f6a892eac1588b346ae02d"
server_key_iv: "3cf4e574f5114af4f306ae6bd19dabbd"
server_key_version: "encryptionkey@0"
I20260620 18:41:37.728883 6403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance:
uuid: "49c4037cd9c6478483d52323e0885eb8"
format_stamp: "Formatted at 2026-06-20 18:41:37 on dist-test-slave-k6lg"
server_key: "29aeb233a6f6a892eac1588b346ae02d"
server_key_iv: "3cf4e574f5114af4f306ae6bd19dabbd"
server_key_version: "encryptionkey@0"
I20260620 18:41:37.734371 6403 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 18:41:37.737716 6418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:37.739136 6403 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:41:37.739269 6403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-2/wal
uuid: "49c4037cd9c6478483d52323e0885eb8"
format_stamp: "Formatted at 2026-06-20 18:41:37 on dist-test-slave-k6lg"
server_key: "29aeb233a6f6a892eac1588b346ae02d"
server_key_iv: "3cf4e574f5114af4f306ae6bd19dabbd"
server_key_version: "encryptionkey@0"
I20260620 18:41:37.739428 6403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:37.760113 6403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:37.764611 6403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:37.764863 6403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:37.765878 6403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:41:37.767170 6403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:37.767246 6403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:37.767325 6403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:37.767369 6403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:37.800709 6403 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.131:34279
I20260620 18:41:37.800761 6530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.131:34279 every 8 connection(s)
I20260620 18:41:37.801964 6403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
I20260620 18:41:37.803249 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 6403
I20260620 18:41:37.803447 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance
I20260620 18:41:37.803960 6022 external_mini_cluster.cc:1428] Setting key 038498198cdc82b8c0eb72a11e40ca07
Jun 20 18:41:37 dist-test-slave-k6lg krb5kdc[6029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980897, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:41:37.821414 6399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:37.804708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:56713 (local address 127.5.225.190:38331)
0620 18:41:37.804889 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:37.804895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:37.806274 (+ 1379us) server_negotiation.cc:407] Connection header received
0620 18:41:37.808431 (+ 2157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:37.808434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:37.808498 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:37.808585 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:37.809993 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:37.810872 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:37.812320 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:37.812697 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:37.812785 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:37.816088 (+ 3303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:37.816093 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:37.816096 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:37.816125 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:37.818590 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:37.819437 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:37.819442 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:37.819444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:37.819581 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:37.820045 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:37.820050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:37.820053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:37.820335 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:37.820447 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:37.821041 (+ 594us) server_negotiation.cc:299] Negotiation successful
0620 18:41:37.821199 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 18:41:37.822428 6533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:37.805234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:38331 (local address 127.5.225.131:56713)
0620 18:41:37.806085 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:37.806137 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:37.808184 (+ 2047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:37.808781 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:37.808789 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:37.809294 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:37.809851 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:37.809867 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:37.811005 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:37.811009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:37.812206 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:37.812217 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:37.812869 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:37.812873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:37.813051 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:37.813778 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:37.813804 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:37.815899 (+ 2095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:37.818717 (+ 2818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:37.818732 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:37.818743 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:37.819312 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:37.819713 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:37.819716 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:37.819719 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:37.819933 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:37.820443 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:37.820448 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:37.820732 (+ 284us) client_negotiation.cc:769] Sending connection context
0620 18:41:37.820954 (+ 222us) client_negotiation.cc:240] Negotiation successful
0620 18:41:37.821312 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":317,"threads_started":1}
I20260620 18:41:37.823769 6531 heartbeater.cc:347] Connected to a master server at 127.5.225.190:38331
I20260620 18:41:37.824079 6531 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:37.824852 6531 heartbeater.cc:511] Master 127.5.225.190:38331 requested a full tablet report, sending...
I20260620 18:41:37.824846 6263 heartbeater.cc:503] Master 127.5.225.190:38331 was elected leader, sending a full tablet report...
I20260620 18:41:37.826447 6073 ts_manager.cc:194] Registered new tserver with Master: 49c4037cd9c6478483d52323e0885eb8 (127.5.225.131:34279)
I20260620 18:41:37.827306 6073 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.131@KRBTEST.COM'} at 127.5.225.131:56713
I20260620 18:41:37.832556 6022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 18:41:37 dist-test-slave-k6lg krb5kdc[6029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980897, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 18:41:37 dist-test-slave-k6lg krb5kdc[6029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980897, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.225.129@KRBTEST.COM
I20260620 18:41:37.878333 6538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:37.862419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47354 (local address 127.5.225.129:44293)
0620 18:41:37.863118 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:37.863128 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:37.863170 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:41:37.863370 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:37.863377 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:37.863627 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:37.863813 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:37.865248 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:37.866343 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:37.867625 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:37.868211 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:37.868353 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:37.872570 (+ 4217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:37.872580 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:37.872595 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:37.872632 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:37.875498 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:37.876177 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:37.876185 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:37.876187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:37.876309 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:37.876715 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:37.876720 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:37.876723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:37.877059 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:37.877354 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:37.877673 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 18:41:37.877938 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":350,"threads_started":1}
W20260620 18:41:37.879609 6178 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47354
Jun 20 18:41:37 dist-test-slave-k6lg krb5kdc[6029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 18:41:37 dist-test-slave-k6lg krb5kdc[6029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.129@KRBTEST.COM
I20260620 18:41:37.921747 6538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:37.908476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47356 (local address 127.5.225.129:44293)
0620 18:41:37.908716 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:37.908722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:37.909064 (+ 342us) server_negotiation.cc:407] Connection header received
0620 18:41:37.909246 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:37.909250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:37.909341 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:37.909438 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:37.910819 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:37.911485 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:37.912664 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:37.913051 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:37.913139 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:37.916971 (+ 3832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:37.916976 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:37.916979 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:37.917013 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:37.919480 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:37.920077 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:37.920081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:37.920083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:37.920195 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:37.920500 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:37.920503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:37.920505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:37.920767 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:37.920929 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:37.921241 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 18:41:37.921392 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 18:41:37.923321 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 6135
I20260620 18:41:37.936946 6538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:37.924402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47360 (local address 127.5.225.129:44293)
0620 18:41:37.924636 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:37.924642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:37.924891 (+ 249us) server_negotiation.cc:407] Connection header received
0620 18:41:37.925027 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:37.925031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:37.925085 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:37.925160 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:37.926625 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:37.927396 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:37.928758 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:37.929185 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:37.929277 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:37.932067 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:37.932072 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:37.932075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:37.932103 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:37.934620 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:37.935263 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:37.935267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:37.935269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:37.935390 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:37.935716 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:37.935719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:37.935721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:37.935952 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:37.936125 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:37.936406 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 18:41:37.936580 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:41:37.937635 6258 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:38.020974 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 6135
I20260620 18:41:38.037190 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 6269
Jun 20 18:41:38 dist-test-slave-k6lg krb5kdc[6029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.130@KRBTEST.COM
I20260620 18:41:38.053453 6545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:38.038082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59896 (local address 127.5.225.130:38703)
0620 18:41:38.038668 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:38.038675 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:38.038702 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:41:38.038762 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:38.038769 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:38.038990 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:38.039113 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:38.040286 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:38.041269 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:38.042601 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:38.043177 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:38.043269 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:38.047363 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:38.047373 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:38.047388 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:38.047421 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:38.050697 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:38.051389 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:38.051397 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:38.051399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:38.051569 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:38.051999 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:38.052002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:38.052004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:38.052338 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:38.052607 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:38.052873 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 18:41:38.053070 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":322,"threads_started":1}
I20260620 18:41:38.054785 6392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:38.137030 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 6269
I20260620 18:41:38.153270 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 6403
Jun 20 18:41:38 dist-test-slave-k6lg krb5kdc[6029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.131@KRBTEST.COM
I20260620 18:41:38.169724 6547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:38.154399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56770 (local address 127.5.225.131:34279)
0620 18:41:38.155142 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:38.155150 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:38.155178 (+ 28us) server_negotiation.cc:407] Connection header received
0620 18:41:38.155239 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:38.155246 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:38.155496 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:38.155631 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:38.156815 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:38.157971 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:38.159364 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:38.159971 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:38.160088 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:38.164167 (+ 4079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:38.164177 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:38.164192 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:38.164231 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:38.167152 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:38.167772 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:38.167779 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:38.167782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:38.167933 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:38.168242 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:38.168245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:38.168247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:38.168569 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:38.168796 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:38.169011 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 18:41:38.169220 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":422,"threads_started":1}
I20260620 18:41:38.171175 6526 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:38.252760 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 6403
I20260620 18:41:38.268548 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 6042
I20260620 18:41:38.269806 6104 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:38.371017 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 6042
2026-06-20T18:41:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3865 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:41:38 dist-test-slave-k6lg krb5kdc[6552](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:41:38 dist-test-slave-k6lg krb5kdc[6552](info): set up 2 sockets
Jun 20 18:41:38 dist-test-slave-k6lg krb5kdc[6552](info): commencing operation
krb5kdc: starting...
W20260620 18:41:39.458069 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:41:39 dist-test-slave-k6lg krb5kdc[6552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:41:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:41:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
I20260620 18:41:39.633764 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:37573
--webserver_interface=127.5.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:44799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:37573
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:39.951212 6565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:39.951529 6565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:39.951691 6565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:39.959048 6565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:41:39.959156 6565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:39.959268 6565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:39.959343 6565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:41:39.959409 6565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:41:39.969249 6565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:44799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:37573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:37573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:41:39.970660 6565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:39.971972 6565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:39.983829 6570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:39.983794 6571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:39.984707 6573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:39.985044 6565 server_base.cc:1034] running on GCE node
I20260620 18:41:39.985826 6565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:39.987383 6565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:39.988627 6565 hybrid_clock.cc:648] HybridClock initialized: now 1781980899988584 us; error 54 us; skew 500 ppm
I20260620 18:41:39.989007 6565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:39 dist-test-slave-k6lg krb5kdc[6552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980899, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:39.992117 6565 init.cc:377] Logged in from keytab as kudu/127.5.225.190@KRBTEST.COM (short username kudu)
I20260620 18:41:39.993484 6565 webserver.cc:460] Webserver started at http://127.5.225.190:40333/ using document root <none> and password file <none>
I20260620 18:41:39.994114 6565 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:39.994268 6565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:39.994531 6565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:39.997373 6565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "36fe008fbc1a423097ee43d02d867af2"
format_stamp: "Formatted at 2026-06-20 18:41:39 on dist-test-slave-k6lg"
server_key: "b28f79b64c0155d32a76b00f1c9dde97"
server_key_iv: "6b2db1556888a87929c20fedc1fc7317"
server_key_version: "encryptionkey@0"
I20260620 18:41:39.998063 6565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "36fe008fbc1a423097ee43d02d867af2"
format_stamp: "Formatted at 2026-06-20 18:41:39 on dist-test-slave-k6lg"
server_key: "b28f79b64c0155d32a76b00f1c9dde97"
server_key_iv: "6b2db1556888a87929c20fedc1fc7317"
server_key_version: "encryptionkey@0"
I20260620 18:41:40.003827 6565 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 18:41:40.007691 6580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:40.009225 6565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:41:40.009398 6565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "36fe008fbc1a423097ee43d02d867af2"
format_stamp: "Formatted at 2026-06-20 18:41:39 on dist-test-slave-k6lg"
server_key: "b28f79b64c0155d32a76b00f1c9dde97"
server_key_iv: "6b2db1556888a87929c20fedc1fc7317"
server_key_version: "encryptionkey@0"
I20260620 18:41:40.009606 6565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:40.043073 6565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:40.047737 6565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:40.047992 6565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:40.066911 6565 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:37573
I20260620 18:41:40.066918 6631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:37573 every 8 connection(s)
I20260620 18:41:40.068302 6565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:41:40.070783 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 6565
I20260620 18:41:40.071012 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:41:40.071661 6022 external_mini_cluster.cc:1428] Setting key 98a5539c662b7ff9005c9a2536b7f4bd
I20260620 18:41:40.072321 6632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:40.087891 6632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36fe008fbc1a423097ee43d02d867af2: Bootstrap starting.
Jun 20 18:41:40 dist-test-slave-k6lg krb5kdc[6552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:41:40.091876 6632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36fe008fbc1a423097ee43d02d867af2: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:40.093220 6632 log.cc:824] T 00000000000000000000000000000000 P 36fe008fbc1a423097ee43d02d867af2: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:40.096956 6632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36fe008fbc1a423097ee43d02d867af2: No bootstrap required, opened a new log
I20260620 18:41:40.098624 6635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:40.073685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43550 (local address 127.5.225.190:37573)
0620 18:41:40.074866 (+ 1181us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:40.074880 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:40.074929 (+ 49us) server_negotiation.cc:407] Connection header received
0620 18:41:40.078201 (+ 3272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:40.078234 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:40.078829 (+ 595us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:40.079324 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:40.080825 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:40.082760 (+ 1935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:40.084081 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:40.085205 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:40.085414 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:40.089573 (+ 4159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:40.089584 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:40.089605 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:40.089662 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:40.093567 (+ 3905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:40.094552 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:40.094559 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:40.094566 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:40.094789 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:40.095334 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:40.095338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:40.095341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:40.096057 (+ 716us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:40.096346 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:40.096758 (+ 412us) server_negotiation.cc:299] Negotiation successful
0620 18:41:40.097212 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":942,"thread_start_us":705,"threads_started":1}
I20260620 18:41:40.105355 6632 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36fe008fbc1a423097ee43d02d867af2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36fe008fbc1a423097ee43d02d867af2" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 37573 } }
I20260620 18:41:40.105819 6632 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36fe008fbc1a423097ee43d02d867af2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:40.105948 6632 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36fe008fbc1a423097ee43d02d867af2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36fe008fbc1a423097ee43d02d867af2, State: Initialized, Role: FOLLOWER
I20260620 18:41:40.106665 6632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36fe008fbc1a423097ee43d02d867af2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36fe008fbc1a423097ee43d02d867af2" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 37573 } }
I20260620 18:41:40.106878 6632 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36fe008fbc1a423097ee43d02d867af2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:41:40.106964 6632 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36fe008fbc1a423097ee43d02d867af2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:41:40.107086 6632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36fe008fbc1a423097ee43d02d867af2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:40.109158 6632 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36fe008fbc1a423097ee43d02d867af2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36fe008fbc1a423097ee43d02d867af2" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 37573 } }
I20260620 18:41:40.109623 6632 leader_election.cc:302] T 00000000000000000000000000000000 P 36fe008fbc1a423097ee43d02d867af2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36fe008fbc1a423097ee43d02d867af2; no voters:
I20260620 18:41:40.110224 6632 leader_election.cc:288] T 00000000000000000000000000000000 P 36fe008fbc1a423097ee43d02d867af2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:41:40.110440 6637 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36fe008fbc1a423097ee43d02d867af2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:41:40.111110 6637 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36fe008fbc1a423097ee43d02d867af2 [term 1 LEADER]: Becoming Leader. State: Replica: 36fe008fbc1a423097ee43d02d867af2, State: Running, Role: LEADER
I20260620 18:41:40.111663 6637 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36fe008fbc1a423097ee43d02d867af2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36fe008fbc1a423097ee43d02d867af2" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 37573 } }
I20260620 18:41:40.112416 6632 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36fe008fbc1a423097ee43d02d867af2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:41:40.114667 6639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36fe008fbc1a423097ee43d02d867af2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36fe008fbc1a423097ee43d02d867af2. Latest consensus state: current_term: 1 leader_uuid: "36fe008fbc1a423097ee43d02d867af2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36fe008fbc1a423097ee43d02d867af2" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 37573 } } }
I20260620 18:41:40.115070 6639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36fe008fbc1a423097ee43d02d867af2 [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:40.115528 6638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36fe008fbc1a423097ee43d02d867af2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36fe008fbc1a423097ee43d02d867af2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36fe008fbc1a423097ee43d02d867af2" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 37573 } } }
I20260620 18:41:40.116539 6638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36fe008fbc1a423097ee43d02d867af2 [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:40.117141 6646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:41:40.128969 6646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:41:40.137835 6646 catalog_manager.cc:1347] Generated new cluster ID: 7ffcc0d0162644beac46495c937c584e
I20260620 18:41:40.137959 6646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:41:40.161278 6646 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:41:40.162863 6646 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:41:40.175346 6646 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36fe008fbc1a423097ee43d02d867af2: Generated new TSK 0
I20260620 18:41:40.176244 6646 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab.
Entry for principal kudu/127.5.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab.
I20260620 18:41:40.225951 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.129:0
--local_ip_for_outbound_sockets=127.5.225.129
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:37573
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:44799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:40.548154 6658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:40.548436 6658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:40.548530 6658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:40.555606 6658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:40.555708 6658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:40.555819 6658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.129
I20260620 18:41:40.565265 6658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:44799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:37573
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.129
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:41:40.566535 6658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:40.567773 6658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:40.580890 6663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:40.580838 6664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:40.580943 6666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:40.581681 6658 server_base.cc:1034] running on GCE node
I20260620 18:41:40.582360 6658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:40.583351 6658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:40.584558 6658 hybrid_clock.cc:648] HybridClock initialized: now 1781980900584511 us; error 37 us; skew 500 ppm
I20260620 18:41:40.584861 6658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:40 dist-test-slave-k6lg krb5kdc[6552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980900, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:40.587990 6658 init.cc:377] Logged in from keytab as kudu/127.5.225.129@KRBTEST.COM (short username kudu)
I20260620 18:41:40.589327 6658 webserver.cc:460] Webserver started at http://127.5.225.129:42841/ using document root <none> and password file <none>
I20260620 18:41:40.589947 6658 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:40.590051 6658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:40.590369 6658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:40.593112 6658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-0/data/instance:
uuid: "f286e9e4b35e4e8fb76ec01092a03719"
format_stamp: "Formatted at 2026-06-20 18:41:40 on dist-test-slave-k6lg"
server_key: "b4ae11750420ecedbf365faa4298ef4f"
server_key_iv: "87b5dc30d7fd37e2fb107e04d7bd20f3"
server_key_version: "encryptionkey@0"
I20260620 18:41:40.593808 6658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance:
uuid: "f286e9e4b35e4e8fb76ec01092a03719"
format_stamp: "Formatted at 2026-06-20 18:41:40 on dist-test-slave-k6lg"
server_key: "b4ae11750420ecedbf365faa4298ef4f"
server_key_iv: "87b5dc30d7fd37e2fb107e04d7bd20f3"
server_key_version: "encryptionkey@0"
I20260620 18:41:40.598989 6658 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:41:40.602430 6673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:40.603829 6658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:41:40.604007 6658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-0/wal
uuid: "f286e9e4b35e4e8fb76ec01092a03719"
format_stamp: "Formatted at 2026-06-20 18:41:40 on dist-test-slave-k6lg"
server_key: "b4ae11750420ecedbf365faa4298ef4f"
server_key_iv: "87b5dc30d7fd37e2fb107e04d7bd20f3"
server_key_version: "encryptionkey@0"
I20260620 18:41:40.604203 6658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:40.623179 6658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:40.627218 6658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:40.627504 6658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:40.628464 6658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:41:40.629751 6658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:40.629855 6658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:40.630008 6658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:40.630081 6658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:40.665006 6658 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.129:43997
I20260620 18:41:40.665094 6785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.129:43997 every 8 connection(s)
I20260620 18:41:40.666343 6658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
I20260620 18:41:40.674561 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 6658
I20260620 18:41:40.674769 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance
I20260620 18:41:40.675406 6022 external_mini_cluster.cc:1428] Setting key 9e843b5f2e0ac6c7951c758068b2c565
Jun 20 18:41:40 dist-test-slave-k6lg krb5kdc[6552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980900, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:41:40.686975 6788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:40.668873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:43505 (local address 127.5.225.190:37573)
0620 18:41:40.669287 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:40.669291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:40.670284 (+ 993us) server_negotiation.cc:407] Connection header received
0620 18:41:40.672093 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:40.672098 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:40.672191 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:40.672306 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:40.673735 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:40.674784 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:40.676717 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:40.677190 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:40.677316 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:40.681444 (+ 4128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:40.681449 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:40.681452 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:40.681489 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:40.683977 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:40.684694 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:40.684700 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:40.684705 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:40.684818 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:40.685247 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:40.685252 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:40.685256 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:40.685531 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:40.685685 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:40.686389 (+ 704us) server_negotiation.cc:299] Negotiation successful
0620 18:41:40.686607 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":212,"threads_started":1}
I20260620 18:41:40.687940 6789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:40.669434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:37573 (local address 127.5.225.129:43505)
0620 18:41:40.670100 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:40.670143 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:40.671823 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:40.672543 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:40.672550 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:40.673039 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:40.673565 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:40.673581 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:40.674970 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:40.674976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:40.676572 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:40.676585 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:40.677447 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:40.677455 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:40.677806 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:40.678539 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:40.678577 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:40.681215 (+ 2638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:40.684126 (+ 2911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:40.684134 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:40.684150 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:40.684562 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:40.684936 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:40.684941 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:40.684944 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:40.685126 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:40.685658 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:40.685666 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:40.685999 (+ 333us) client_negotiation.cc:769] Sending connection context
0620 18:41:40.686323 (+ 324us) client_negotiation.cc:240] Negotiation successful
0620 18:41:40.686857 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":256,"threads_started":1}
I20260620 18:41:40.689520 6786 heartbeater.cc:347] Connected to a master server at 127.5.225.190:37573
I20260620 18:41:40.689854 6786 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:40.690776 6786 heartbeater.cc:511] Master 127.5.225.190:37573 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.225.130@KRBTEST.COM; defaulting to no policy
I20260620 18:41:40.692997 6597 ts_manager.cc:194] Registered new tserver with Master: f286e9e4b35e4e8fb76ec01092a03719 (127.5.225.129:43997)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.130@KRBTEST.COM" created.
I20260620 18:41:40.694768 6597 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.129@KRBTEST.COM'} at 127.5.225.129:43505
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab.
Entry for principal kudu/127.5.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab.
I20260620 18:41:40.713766 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.130:0
--local_ip_for_outbound_sockets=127.5.225.130
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:37573
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:44799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:41.026698 6792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:41.026981 6792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:41.027165 6792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:41.034880 6792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:41.035007 6792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:41.035187 6792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.130
I20260620 18:41:41.045135 6792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:44799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:37573
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.130
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:41:41.046525 6792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:41.047780 6792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:41.060331 6797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:41.060398 6800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:41.060437 6798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:41.061292 6792 server_base.cc:1034] running on GCE node
I20260620 18:41:41.061893 6792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:41.062860 6792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:41.064073 6792 hybrid_clock.cc:648] HybridClock initialized: now 1781980901064004 us; error 47 us; skew 500 ppm
I20260620 18:41:41.064395 6792 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:41 dist-test-slave-k6lg krb5kdc[6552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980901, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:41.067468 6792 init.cc:377] Logged in from keytab as kudu/127.5.225.130@KRBTEST.COM (short username kudu)
I20260620 18:41:41.069038 6792 webserver.cc:460] Webserver started at http://127.5.225.130:42693/ using document root <none> and password file <none>
I20260620 18:41:41.069712 6792 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:41.069835 6792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:41.070137 6792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:41.072862 6792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-1/data/instance:
uuid: "fc1c3b0211b64ba982b13f0a233a9e3d"
format_stamp: "Formatted at 2026-06-20 18:41:41 on dist-test-slave-k6lg"
server_key: "8f61df9a341e05c30fbbb610dc450655"
server_key_iv: "b0a3a5ebf04c0ae1f748d79095022a11"
server_key_version: "encryptionkey@0"
I20260620 18:41:41.073627 6792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance:
uuid: "fc1c3b0211b64ba982b13f0a233a9e3d"
format_stamp: "Formatted at 2026-06-20 18:41:41 on dist-test-slave-k6lg"
server_key: "8f61df9a341e05c30fbbb610dc450655"
server_key_iv: "b0a3a5ebf04c0ae1f748d79095022a11"
server_key_version: "encryptionkey@0"
I20260620 18:41:41.078977 6792 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260620 18:41:41.083375 6807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:41.084723 6792 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260620 18:41:41.084853 6792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-1/wal
uuid: "fc1c3b0211b64ba982b13f0a233a9e3d"
format_stamp: "Formatted at 2026-06-20 18:41:41 on dist-test-slave-k6lg"
server_key: "8f61df9a341e05c30fbbb610dc450655"
server_key_iv: "b0a3a5ebf04c0ae1f748d79095022a11"
server_key_version: "encryptionkey@0"
I20260620 18:41:41.085013 6792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:41.114485 6792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:41.119447 6792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:41.119741 6792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:41.120749 6792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:41:41.122382 6792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:41.122470 6792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:41.122587 6792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:41.122638 6792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:41.156494 6792 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.130:35693
I20260620 18:41:41.156536 6919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.130:35693 every 8 connection(s)
I20260620 18:41:41.157763 6792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
I20260620 18:41:41.160333 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 6792
I20260620 18:41:41.160548 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance
I20260620 18:41:41.161027 6022 external_mini_cluster.cc:1428] Setting key a54bf5b01e342fe925919c3af66f2c7f
Jun 20 18:41:41 dist-test-slave-k6lg krb5kdc[6552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980901, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.225.131@KRBTEST.COM; defaulting to no policy
I20260620 18:41:41.178301 6788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:41.160453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:56809 (local address 127.5.225.190:37573)
0620 18:41:41.160700 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:41.160709 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:41.161941 (+ 1232us) server_negotiation.cc:407] Connection header received
0620 18:41:41.163749 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:41.163757 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:41.163842 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:41.163949 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:41.165378 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:41.166563 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:41.168073 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:41.168457 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:41.168596 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:41.172250 (+ 3654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:41.172255 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:41.172259 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:41.172290 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:41.175046 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:41.175925 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:41.175932 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:41.175936 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:41.176076 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:41.176491 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:41.176497 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:41.176501 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:41.176759 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:41.176921 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:41.177697 (+ 776us) server_negotiation.cc:299] Negotiation successful
0620 18:41:41.177914 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260620 18:41:41.179291 6922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:41.160955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:37573 (local address 127.5.225.130:56809)
0620 18:41:41.161773 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:41.161825 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:41.163487 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:41.164149 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:41.164157 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:41.164665 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:41.165204 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:41.165221 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:41.166705 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:41.166708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:41.167915 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:41.167926 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:41.168695 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:41.168704 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:41.168932 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:41.169664 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:41.169685 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:41.172030 (+ 2345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:41.175217 (+ 3187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:41.175225 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:41.175243 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:41.175800 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:41.176187 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:41.176192 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:41.176195 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:41.176378 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:41.176908 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:41.176952 (+ 44us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:41.177333 (+ 381us) client_negotiation.cc:769] Sending connection context
0620 18:41:41.177614 (+ 281us) client_negotiation.cc:240] Negotiation successful
0620 18:41:41.178128 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":336,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.131@KRBTEST.COM" created.
I20260620 18:41:41.180721 6920 heartbeater.cc:347] Connected to a master server at 127.5.225.190:37573
I20260620 18:41:41.181041 6920 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:41.181897 6920 heartbeater.cc:511] Master 127.5.225.190:37573 requested a full tablet report, sending...
I20260620 18:41:41.183439 6597 ts_manager.cc:194] Registered new tserver with Master: fc1c3b0211b64ba982b13f0a233a9e3d (127.5.225.130:35693)
I20260620 18:41:41.184226 6597 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.130@KRBTEST.COM'} at 127.5.225.130:56809
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab.
Entry for principal kudu/127.5.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab.
I20260620 18:41:41.199388 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.131:0
--local_ip_for_outbound_sockets=127.5.225.131
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:37573
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:44799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:41.517442 6925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:41.517760 6925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:41.517935 6925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:41.525203 6925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:41.525352 6925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:41.525547 6925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.131
I20260620 18:41:41.535241 6925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:44799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:37573
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.131
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:41:41.536695 6925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:41.538440 6925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:41.551615 6931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:41.551703 6933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:41.551666 6930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:41.552551 6925 server_base.cc:1034] running on GCE node
I20260620 18:41:41.553241 6925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:41.554133 6925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:41.555390 6925 hybrid_clock.cc:648] HybridClock initialized: now 1781980901555301 us; error 81 us; skew 500 ppm
I20260620 18:41:41.555719 6925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:41 dist-test-slave-k6lg krb5kdc[6552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980901, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:41.558984 6925 init.cc:377] Logged in from keytab as kudu/127.5.225.131@KRBTEST.COM (short username kudu)
I20260620 18:41:41.560461 6925 webserver.cc:460] Webserver started at http://127.5.225.131:40893/ using document root <none> and password file <none>
I20260620 18:41:41.561144 6925 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:41.561282 6925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:41.561591 6925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:41.564492 6925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-2/data/instance:
uuid: "31487f8a81dd4a1ba7d0772d4c18fc3a"
format_stamp: "Formatted at 2026-06-20 18:41:41 on dist-test-slave-k6lg"
server_key: "defc3d570000a22d0880ecde767702aa"
server_key_iv: "88d613a07f727a4019dd9beb24377797"
server_key_version: "encryptionkey@0"
I20260620 18:41:41.565258 6925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance:
uuid: "31487f8a81dd4a1ba7d0772d4c18fc3a"
format_stamp: "Formatted at 2026-06-20 18:41:41 on dist-test-slave-k6lg"
server_key: "defc3d570000a22d0880ecde767702aa"
server_key_iv: "88d613a07f727a4019dd9beb24377797"
server_key_version: "encryptionkey@0"
I20260620 18:41:41.570050 6925 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260620 18:41:41.573349 6940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:41.574674 6925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:41:41.574844 6925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-2/wal
uuid: "31487f8a81dd4a1ba7d0772d4c18fc3a"
format_stamp: "Formatted at 2026-06-20 18:41:41 on dist-test-slave-k6lg"
server_key: "defc3d570000a22d0880ecde767702aa"
server_key_iv: "88d613a07f727a4019dd9beb24377797"
server_key_version: "encryptionkey@0"
I20260620 18:41:41.575037 6925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:41.594273 6925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:41.598279 6925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:41.598511 6925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:41.599418 6925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:41:41.600653 6925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:41.600720 6925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:41.600802 6925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:41.600847 6925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:41.633893 6925 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.131:40417
I20260620 18:41:41.633975 7052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.131:40417 every 8 connection(s)
I20260620 18:41:41.635222 6925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
I20260620 18:41:41.645018 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 6925
I20260620 18:41:41.645258 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance
I20260620 18:41:41.645706 6022 external_mini_cluster.cc:1428] Setting key f4d6177d2a2a880722aac6f45c5d2880
Jun 20 18:41:41 dist-test-slave-k6lg krb5kdc[6552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980901, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:41:41.656067 6788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:41.637658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:56063 (local address 127.5.225.190:37573)
0620 18:41:41.637869 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:41.637876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:41.639070 (+ 1194us) server_negotiation.cc:407] Connection header received
0620 18:41:41.640843 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:41.640847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:41.640912 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:41.641003 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:41.642616 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:41.643349 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:41.645153 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:41.645609 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:41.645700 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:41.649734 (+ 4034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:41.649739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:41.649742 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:41.649775 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:41.652845 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:41.653709 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:41.653714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:41.653717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:41.653846 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:41.654361 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:41.654364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:41.654366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:41.654640 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:41.654750 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:41.655524 (+ 774us) server_negotiation.cc:299] Negotiation successful
0620 18:41:41.655778 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 18:41:41.656932 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:41.638212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:37573 (local address 127.5.225.131:56063)
0620 18:41:41.638906 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:41.638948 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:41.640595 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:41.641240 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:41.641249 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:41.641805 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:41.642443 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:41.642464 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:41.643539 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:41.643543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:41.644965 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:41.644982 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:41.645838 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:41.645846 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:41.646132 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:41.646877 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:41.646903 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:41.649532 (+ 2629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:41.653027 (+ 3495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:41.653037 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:41.653055 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:41.653583 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:41.653977 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:41.653981 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:41.653984 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:41.654244 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:41.654752 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:41.654758 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:41.655105 (+ 347us) client_negotiation.cc:769] Sending connection context
0620 18:41:41.655342 (+ 237us) client_negotiation.cc:240] Negotiation successful
0620 18:41:41.655766 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":262,"threads_started":1}
I20260620 18:41:41.658520 7053 heartbeater.cc:347] Connected to a master server at 127.5.225.190:37573
I20260620 18:41:41.658825 7053 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:41.659691 7053 heartbeater.cc:511] Master 127.5.225.190:37573 requested a full tablet report, sending...
I20260620 18:41:41.661494 6596 ts_manager.cc:194] Registered new tserver with Master: 31487f8a81dd4a1ba7d0772d4c18fc3a (127.5.225.131:40417)
I20260620 18:41:41.662447 6596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.131@KRBTEST.COM'} at 127.5.225.131:56063
I20260620 18:41:41.673821 6022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:41:41.693116 6788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:41.680495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43556 (local address 127.5.225.190:37573)
0620 18:41:41.680737 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:41.680746 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:41.681099 (+ 353us) server_negotiation.cc:407] Connection header received
0620 18:41:41.681281 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:41.681286 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:41.681347 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:41.681434 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:41.682990 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:41.683754 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:41.685009 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:41.685401 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:41.685495 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:41.687880 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:41.687884 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:41.687887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:41.687918 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:41.690701 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:41.691337 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:41.691341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:41.691344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:41.691464 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:41.691785 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:41.691788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:41.691790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:41.692085 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:41.692254 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:41.692590 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 18:41:41.692739 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:41:41.697603 6786 heartbeater.cc:503] Master 127.5.225.190:37573 was elected leader, sending a full tablet report...
I20260620 18:41:41.699944 6596 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:41:41.707425 6596 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:41:41.736270 7067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:41.720472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43518 (local address 127.5.225.131:40417)
0620 18:41:41.728536 (+ 8064us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:41.728546 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:41.728576 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:41:41.728662 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:41.728670 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:41.728884 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:41.729086 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:41.729723 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:41.731317 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:41.733311 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:41.734715 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:41.734843 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:41.735124 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:41.735739 (+ 615us) server_negotiation.cc:299] Negotiation successful
0620 18:41:41.735910 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7905,"thread_start_us":7832,"threads_started":1}
I20260620 18:41:41.736380 7065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:41.720187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.131:40417 (local address 127.0.0.1:43518)
0620 18:41:41.721169 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:41.721217 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:41.721436 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:41.729075 (+ 7639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:41.729080 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:41.729108 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:41.729535 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:41.729547 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:41.731457 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:41.731466 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:41.733165 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:41.733176 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:41.735270 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:41.735276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:41.735482 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:41.735553 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 18:41:41.735682 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 18:41:41.735910 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"mutex_wait_us":99,"thread_start_us":927,"threads_started":1}
I20260620 18:41:41.736599 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:41.722444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48484 (local address 127.5.225.130:35693)
0620 18:41:41.726706 (+ 4262us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:41.726719 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:41.727844 (+ 1125us) server_negotiation.cc:407] Connection header received
0620 18:41:41.728104 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:41.728112 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:41.728328 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:41.728544 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:41.729240 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:41.730721 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:41.733576 (+ 2855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:41.734980 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:41.735130 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:41.735323 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:41.736099 (+ 776us) server_negotiation.cc:299] Negotiation successful
0620 18:41:41.736262 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4099,"thread_start_us":4003,"threads_started":1}
I20260620 18:41:41.736416 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:41.720422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:43997 (local address 127.0.0.1:53498)
0620 18:41:41.722403 (+ 1981us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:41.722421 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:41.722548 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:41.728601 (+ 6053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:41.728607 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:41.728632 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:41.729186 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:41.729200 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:41.731407 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:41.731415 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:41.733722 (+ 2307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:41.733731 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:41.735473 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:41.735478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:41.735662 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:41.735691 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 18:41:41.735742 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 18:41:41.735920 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1846,"mutex_wait_us":148,"thread_start_us":6970,"threads_started":1}
I20260620 18:41:41.737708 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:41.720891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53498 (local address 127.5.225.129:43997)
0620 18:41:41.727327 (+ 6436us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:41.727339 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:41.727373 (+ 34us) server_negotiation.cc:407] Connection header received
0620 18:41:41.727496 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:41.727504 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:41.727779 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:41.729413 (+ 1634us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:41.729604 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:41.731248 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:41.733849 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:41.735235 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:41.736965 (+ 1730us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:41.737191 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:41.737254 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 18:41:41.737396 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6223,"thread_start_us":6051,"threads_started":1}
I20260620 18:41:41.738543 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:41.721866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.130:35693 (local address 127.0.0.1:48484)
0620 18:41:41.727724 (+ 5858us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:41.727747 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:41.727910 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:41.728547 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:41.728556 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:41.728592 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:41.729023 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:41.729041 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:41.730868 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:41.730873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:41.733410 (+ 2537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:41.733429 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:41.735204 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:41.735211 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:41.735441 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:41.735574 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 18:41:41.735662 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 18:41:41.735911 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5698,"mutex_wait_us":250,"thread_start_us":772,"threads_started":1}
I20260620 18:41:41.740818 6988 tablet_service.cc:1467] Processing CreateTablet for tablet 718e9b57b4514963ab272ee0540ae8b8 (DEFAULT_TABLE table=test-table [id=055e988cb92d4fa59894ccdc1d7813ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:41:41.742410 6988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 718e9b57b4514963ab272ee0540ae8b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:41.742050 6855 tablet_service.cc:1467] Processing CreateTablet for tablet 718e9b57b4514963ab272ee0540ae8b8 (DEFAULT_TABLE table=test-table [id=055e988cb92d4fa59894ccdc1d7813ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:41:41.743372 6855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 718e9b57b4514963ab272ee0540ae8b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:41.744179 6721 tablet_service.cc:1467] Processing CreateTablet for tablet 718e9b57b4514963ab272ee0540ae8b8 (DEFAULT_TABLE table=test-table [id=055e988cb92d4fa59894ccdc1d7813ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:41:41.745482 6721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 718e9b57b4514963ab272ee0540ae8b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:41.757974 7072 tablet_bootstrap.cc:492] T 718e9b57b4514963ab272ee0540ae8b8 P 31487f8a81dd4a1ba7d0772d4c18fc3a: Bootstrap starting.
I20260620 18:41:41.759454 7071 tablet_bootstrap.cc:492] T 718e9b57b4514963ab272ee0540ae8b8 P f286e9e4b35e4e8fb76ec01092a03719: Bootstrap starting.
I20260620 18:41:41.761251 7073 tablet_bootstrap.cc:492] T 718e9b57b4514963ab272ee0540ae8b8 P fc1c3b0211b64ba982b13f0a233a9e3d: Bootstrap starting.
I20260620 18:41:41.762480 7072 tablet_bootstrap.cc:654] T 718e9b57b4514963ab272ee0540ae8b8 P 31487f8a81dd4a1ba7d0772d4c18fc3a: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:41.762969 7071 tablet_bootstrap.cc:654] T 718e9b57b4514963ab272ee0540ae8b8 P f286e9e4b35e4e8fb76ec01092a03719: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:41.763926 7071 log.cc:824] T 718e9b57b4514963ab272ee0540ae8b8 P f286e9e4b35e4e8fb76ec01092a03719: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:41.764024 7072 log.cc:824] T 718e9b57b4514963ab272ee0540ae8b8 P 31487f8a81dd4a1ba7d0772d4c18fc3a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:41.765043 7073 tablet_bootstrap.cc:654] T 718e9b57b4514963ab272ee0540ae8b8 P fc1c3b0211b64ba982b13f0a233a9e3d: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:41.766727 7073 log.cc:824] T 718e9b57b4514963ab272ee0540ae8b8 P fc1c3b0211b64ba982b13f0a233a9e3d: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:41.766810 7071 tablet_bootstrap.cc:492] T 718e9b57b4514963ab272ee0540ae8b8 P f286e9e4b35e4e8fb76ec01092a03719: No bootstrap required, opened a new log
I20260620 18:41:41.767068 7071 ts_tablet_manager.cc:1397] T 718e9b57b4514963ab272ee0540ae8b8 P f286e9e4b35e4e8fb76ec01092a03719: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 18:41:41.769586 7073 tablet_bootstrap.cc:492] T 718e9b57b4514963ab272ee0540ae8b8 P fc1c3b0211b64ba982b13f0a233a9e3d: No bootstrap required, opened a new log
I20260620 18:41:41.769869 7073 ts_tablet_manager.cc:1397] T 718e9b57b4514963ab272ee0540ae8b8 P fc1c3b0211b64ba982b13f0a233a9e3d: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 18:41:41.770129 7072 tablet_bootstrap.cc:492] T 718e9b57b4514963ab272ee0540ae8b8 P 31487f8a81dd4a1ba7d0772d4c18fc3a: No bootstrap required, opened a new log
I20260620 18:41:41.770409 7072 ts_tablet_manager.cc:1397] T 718e9b57b4514963ab272ee0540ae8b8 P 31487f8a81dd4a1ba7d0772d4c18fc3a: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.005s
I20260620 18:41:41.777616 7071 raft_consensus.cc:348] T 718e9b57b4514963ab272ee0540ae8b8 P f286e9e4b35e4e8fb76ec01092a03719 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31487f8a81dd4a1ba7d0772d4c18fc3a" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 40417 } } peers { permanent_uuid: "f286e9e4b35e4e8fb76ec01092a03719" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 43997 } } peers { permanent_uuid: "fc1c3b0211b64ba982b13f0a233a9e3d" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 35693 } }
I20260620 18:41:41.778043 7071 raft_consensus.cc:374] T 718e9b57b4514963ab272ee0540ae8b8 P f286e9e4b35e4e8fb76ec01092a03719 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:41.778334 7071 raft_consensus.cc:729] T 718e9b57b4514963ab272ee0540ae8b8 P f286e9e4b35e4e8fb76ec01092a03719 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f286e9e4b35e4e8fb76ec01092a03719, State: Initialized, Role: FOLLOWER
I20260620 18:41:41.778317 7073 raft_consensus.cc:348] T 718e9b57b4514963ab272ee0540ae8b8 P fc1c3b0211b64ba982b13f0a233a9e3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31487f8a81dd4a1ba7d0772d4c18fc3a" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 40417 } } peers { permanent_uuid: "f286e9e4b35e4e8fb76ec01092a03719" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 43997 } } peers { permanent_uuid: "fc1c3b0211b64ba982b13f0a233a9e3d" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 35693 } }
I20260620 18:41:41.778769 7073 raft_consensus.cc:374] T 718e9b57b4514963ab272ee0540ae8b8 P fc1c3b0211b64ba982b13f0a233a9e3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:41.778913 7073 raft_consensus.cc:729] T 718e9b57b4514963ab272ee0540ae8b8 P fc1c3b0211b64ba982b13f0a233a9e3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc1c3b0211b64ba982b13f0a233a9e3d, State: Initialized, Role: FOLLOWER
I20260620 18:41:41.779067 7071 consensus_queue.cc:260] T 718e9b57b4514963ab272ee0540ae8b8 P f286e9e4b35e4e8fb76ec01092a03719 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31487f8a81dd4a1ba7d0772d4c18fc3a" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 40417 } } peers { permanent_uuid: "f286e9e4b35e4e8fb76ec01092a03719" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 43997 } } peers { permanent_uuid: "fc1c3b0211b64ba982b13f0a233a9e3d" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 35693 } }
I20260620 18:41:41.779624 7073 consensus_queue.cc:260] T 718e9b57b4514963ab272ee0540ae8b8 P fc1c3b0211b64ba982b13f0a233a9e3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31487f8a81dd4a1ba7d0772d4c18fc3a" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 40417 } } peers { permanent_uuid: "f286e9e4b35e4e8fb76ec01092a03719" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 43997 } } peers { permanent_uuid: "fc1c3b0211b64ba982b13f0a233a9e3d" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 35693 } }
I20260620 18:41:41.781044 7071 ts_tablet_manager.cc:1428] T 718e9b57b4514963ab272ee0540ae8b8 P f286e9e4b35e4e8fb76ec01092a03719: Time spent starting tablet: real 0.014s user 0.008s sys 0.005s
I20260620 18:41:41.780877 7072 raft_consensus.cc:348] T 718e9b57b4514963ab272ee0540ae8b8 P 31487f8a81dd4a1ba7d0772d4c18fc3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31487f8a81dd4a1ba7d0772d4c18fc3a" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 40417 } } peers { permanent_uuid: "f286e9e4b35e4e8fb76ec01092a03719" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 43997 } } peers { permanent_uuid: "fc1c3b0211b64ba982b13f0a233a9e3d" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 35693 } }
I20260620 18:41:41.781627 7072 raft_consensus.cc:374] T 718e9b57b4514963ab272ee0540ae8b8 P 31487f8a81dd4a1ba7d0772d4c18fc3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:41.781739 7072 raft_consensus.cc:729] T 718e9b57b4514963ab272ee0540ae8b8 P 31487f8a81dd4a1ba7d0772d4c18fc3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31487f8a81dd4a1ba7d0772d4c18fc3a, State: Initialized, Role: FOLLOWER
I20260620 18:41:41.782351 7072 consensus_queue.cc:260] T 718e9b57b4514963ab272ee0540ae8b8 P 31487f8a81dd4a1ba7d0772d4c18fc3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31487f8a81dd4a1ba7d0772d4c18fc3a" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 40417 } } peers { permanent_uuid: "f286e9e4b35e4e8fb76ec01092a03719" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 43997 } } peers { permanent_uuid: "fc1c3b0211b64ba982b13f0a233a9e3d" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 35693 } }
I20260620 18:41:41.782850 6920 heartbeater.cc:503] Master 127.5.225.190:37573 was elected leader, sending a full tablet report...
I20260620 18:41:41.783578 7073 ts_tablet_manager.cc:1428] T 718e9b57b4514963ab272ee0540ae8b8 P fc1c3b0211b64ba982b13f0a233a9e3d: Time spent starting tablet: real 0.014s user 0.008s sys 0.006s
I20260620 18:41:41.784978 7072 ts_tablet_manager.cc:1428] T 718e9b57b4514963ab272ee0540ae8b8 P 31487f8a81dd4a1ba7d0772d4c18fc3a: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260620 18:41:41.785636 7053 heartbeater.cc:503] Master 127.5.225.190:37573 was elected leader, sending a full tablet report...
W20260620 18:41:41.887100 7054 tablet.cc:2369] T 718e9b57b4514963ab272ee0540ae8b8 P 31487f8a81dd4a1ba7d0772d4c18fc3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:41:41.909935 6921 tablet.cc:2369] T 718e9b57b4514963ab272ee0540ae8b8 P fc1c3b0211b64ba982b13f0a233a9e3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:41:41.918676 6787 tablet.cc:2369] T 718e9b57b4514963ab272ee0540ae8b8 P f286e9e4b35e4e8fb76ec01092a03719: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:41:41.988940 7077 raft_consensus.cc:482] T 718e9b57b4514963ab272ee0540ae8b8 P f286e9e4b35e4e8fb76ec01092a03719 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:41:41.989293 7077 raft_consensus.cc:504] T 718e9b57b4514963ab272ee0540ae8b8 P f286e9e4b35e4e8fb76ec01092a03719 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31487f8a81dd4a1ba7d0772d4c18fc3a" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 40417 } } peers { permanent_uuid: "f286e9e4b35e4e8fb76ec01092a03719" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 43997 } } peers { permanent_uuid: "fc1c3b0211b64ba982b13f0a233a9e3d" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 35693 } }
I20260620 18:41:41.991137 7077 leader_election.cc:288] T 718e9b57b4514963ab272ee0540ae8b8 P f286e9e4b35e4e8fb76ec01092a03719 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31487f8a81dd4a1ba7d0772d4c18fc3a (127.5.225.131:40417), fc1c3b0211b64ba982b13f0a233a9e3d (127.5.225.130:35693)
I20260620 18:41:41.998270 7067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:41.991649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:53985 (local address 127.5.225.131:40417)
0620 18:41:41.991946 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:41.991957 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:41.992381 (+ 424us) server_negotiation.cc:407] Connection header received
0620 18:41:41.992880 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:41.992884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:41.992978 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:41.993102 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:41.993700 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:41.994494 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:41.996468 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:41.997240 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:41.997387 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:41.997471 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:41.997851 (+ 380us) server_negotiation.cc:299] Negotiation successful
0620 18:41:41.997977 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260620 18:41:41.998317 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:41.991687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:36625 (local address 127.5.225.130:35693)
0620 18:41:41.992034 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:41.992042 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:41.992573 (+ 531us) server_negotiation.cc:407] Connection header received
0620 18:41:41.992745 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:41.992753 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:41.992843 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:41.992924 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:41.993652 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:41.994512 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:41.996510 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:41.997275 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:41.997463 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:41.997533 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:41.997904 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 18:41:41.998031 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 18:41:41.998404 7080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:41.991441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.131:40417 (local address 127.5.225.129:53985)
0620 18:41:41.992247 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:41.992281 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:41.992713 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:41.993137 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:41.993140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:41.993171 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:41.993535 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:41.993544 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:41.994637 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:41.994644 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:41.996312 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:41.996326 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:41.997472 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:41.997478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:41.997680 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:41.997719 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 18:41:41.997790 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 18:41:41.997925 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"mutex_wait_us":90,"thread_start_us":546,"threads_started":1}
I20260620 18:41:41.998448 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:41.991638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.130:35693 (local address 127.5.225.129:36625)
0620 18:41:41.992462 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:41.992476 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:41.992593 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:41.992968 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:41.992972 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:41.992996 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:41.993457 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:41.993466 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:41.994662 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:41.994665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:41.996331 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:41.996342 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:41.997532 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:41.997537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:41.997726 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:41.997748 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 18:41:41.997857 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 18:41:41.997976 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"mutex_wait_us":68,"spinlock_wait_cycles":28928,"thread_start_us":518,"threads_started":1}
I20260620 18:41:41.999370 6875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718e9b57b4514963ab272ee0540ae8b8" candidate_uuid: "f286e9e4b35e4e8fb76ec01092a03719" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc1c3b0211b64ba982b13f0a233a9e3d" is_pre_election: true
I20260620 18:41:41.999416 7008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718e9b57b4514963ab272ee0540ae8b8" candidate_uuid: "f286e9e4b35e4e8fb76ec01092a03719" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31487f8a81dd4a1ba7d0772d4c18fc3a" is_pre_election: true
I20260620 18:41:41.999867 6875 raft_consensus.cc:2454] T 718e9b57b4514963ab272ee0540ae8b8 P fc1c3b0211b64ba982b13f0a233a9e3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f286e9e4b35e4e8fb76ec01092a03719 in term 0.
I20260620 18:41:41.999888 7008 raft_consensus.cc:2454] T 718e9b57b4514963ab272ee0540ae8b8 P 31487f8a81dd4a1ba7d0772d4c18fc3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f286e9e4b35e4e8fb76ec01092a03719 in term 0.
I20260620 18:41:42.000520 6675 leader_election.cc:302] T 718e9b57b4514963ab272ee0540ae8b8 P f286e9e4b35e4e8fb76ec01092a03719 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 31487f8a81dd4a1ba7d0772d4c18fc3a, f286e9e4b35e4e8fb76ec01092a03719, fc1c3b0211b64ba982b13f0a233a9e3d; no voters:
I20260620 18:41:42.000932 7077 raft_consensus.cc:2781] T 718e9b57b4514963ab272ee0540ae8b8 P f286e9e4b35e4e8fb76ec01092a03719 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:41:42.001088 7077 raft_consensus.cc:482] T 718e9b57b4514963ab272ee0540ae8b8 P f286e9e4b35e4e8fb76ec01092a03719 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:41:42.001206 7077 raft_consensus.cc:3037] T 718e9b57b4514963ab272ee0540ae8b8 P f286e9e4b35e4e8fb76ec01092a03719 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:42.003448 7077 raft_consensus.cc:504] T 718e9b57b4514963ab272ee0540ae8b8 P f286e9e4b35e4e8fb76ec01092a03719 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31487f8a81dd4a1ba7d0772d4c18fc3a" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 40417 } } peers { permanent_uuid: "f286e9e4b35e4e8fb76ec01092a03719" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 43997 } } peers { permanent_uuid: "fc1c3b0211b64ba982b13f0a233a9e3d" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 35693 } }
I20260620 18:41:42.004047 7077 leader_election.cc:288] T 718e9b57b4514963ab272ee0540ae8b8 P f286e9e4b35e4e8fb76ec01092a03719 [CANDIDATE]: Term 1 election: Requested vote from peers 31487f8a81dd4a1ba7d0772d4c18fc3a (127.5.225.131:40417), fc1c3b0211b64ba982b13f0a233a9e3d (127.5.225.130:35693)
I20260620 18:41:42.004544 6875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718e9b57b4514963ab272ee0540ae8b8" candidate_uuid: "f286e9e4b35e4e8fb76ec01092a03719" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc1c3b0211b64ba982b13f0a233a9e3d"
I20260620 18:41:42.004563 7008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718e9b57b4514963ab272ee0540ae8b8" candidate_uuid: "f286e9e4b35e4e8fb76ec01092a03719" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31487f8a81dd4a1ba7d0772d4c18fc3a"
I20260620 18:41:42.004740 6875 raft_consensus.cc:3037] T 718e9b57b4514963ab272ee0540ae8b8 P fc1c3b0211b64ba982b13f0a233a9e3d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:42.004763 7008 raft_consensus.cc:3037] T 718e9b57b4514963ab272ee0540ae8b8 P 31487f8a81dd4a1ba7d0772d4c18fc3a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:42.006888 6875 raft_consensus.cc:2454] T 718e9b57b4514963ab272ee0540ae8b8 P fc1c3b0211b64ba982b13f0a233a9e3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f286e9e4b35e4e8fb76ec01092a03719 in term 1.
I20260620 18:41:42.007047 7008 raft_consensus.cc:2454] T 718e9b57b4514963ab272ee0540ae8b8 P 31487f8a81dd4a1ba7d0772d4c18fc3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f286e9e4b35e4e8fb76ec01092a03719 in term 1.
I20260620 18:41:42.007313 6675 leader_election.cc:302] T 718e9b57b4514963ab272ee0540ae8b8 P f286e9e4b35e4e8fb76ec01092a03719 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f286e9e4b35e4e8fb76ec01092a03719, fc1c3b0211b64ba982b13f0a233a9e3d; no voters:
I20260620 18:41:42.007604 7077 raft_consensus.cc:2781] T 718e9b57b4514963ab272ee0540ae8b8 P f286e9e4b35e4e8fb76ec01092a03719 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:41:42.008081 7077 raft_consensus.cc:686] T 718e9b57b4514963ab272ee0540ae8b8 P f286e9e4b35e4e8fb76ec01092a03719 [term 1 LEADER]: Becoming Leader. State: Replica: f286e9e4b35e4e8fb76ec01092a03719, State: Running, Role: LEADER
I20260620 18:41:42.008646 7077 consensus_queue.cc:237] T 718e9b57b4514963ab272ee0540ae8b8 P f286e9e4b35e4e8fb76ec01092a03719 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31487f8a81dd4a1ba7d0772d4c18fc3a" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 40417 } } peers { permanent_uuid: "f286e9e4b35e4e8fb76ec01092a03719" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 43997 } } peers { permanent_uuid: "fc1c3b0211b64ba982b13f0a233a9e3d" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 35693 } }
I20260620 18:41:42.013106 6597 catalog_manager.cc:5495] T 718e9b57b4514963ab272ee0540ae8b8 P f286e9e4b35e4e8fb76ec01092a03719 reported cstate change: term changed from 0 to 1, leader changed from <none> to f286e9e4b35e4e8fb76ec01092a03719 (127.5.225.129). New cstate: current_term: 1 leader_uuid: "f286e9e4b35e4e8fb76ec01092a03719" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31487f8a81dd4a1ba7d0772d4c18fc3a" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 40417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f286e9e4b35e4e8fb76ec01092a03719" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 43997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc1c3b0211b64ba982b13f0a233a9e3d" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 35693 } health_report { overall_health: UNKNOWN } } }
Jun 20 18:41:42 dist-test-slave-k6lg krb5kdc[6552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.129@KRBTEST.COM
I20260620 18:41:42.036917 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:42.022128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53512 (local address 127.5.225.129:43997)
0620 18:41:42.022346 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:42.022351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:42.022909 (+ 558us) server_negotiation.cc:407] Connection header received
0620 18:41:42.023145 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:42.023150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:42.023212 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:42.023314 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:42.024762 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:42.025468 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:42.026791 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:42.027209 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:42.027340 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:42.031327 (+ 3987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:42.031340 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:42.031356 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:42.031396 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:42.034142 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:42.034868 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:42.034876 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:42.034878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:42.035004 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:42.035388 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:42.035392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:42.035394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:42.035745 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:42.036047 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:42.036341 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 18:41:42.036596 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
Jun 20 18:41:42 dist-test-slave-k6lg krb5kdc[6552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980902, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 18:41:42 dist-test-slave-k6lg krb5kdc[6552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980902, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.225.129@KRBTEST.COM
I20260620 18:41:42.083644 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:42.067909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53514 (local address 127.5.225.129:43997)
0620 18:41:42.068112 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:42.068120 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:42.068476 (+ 356us) server_negotiation.cc:407] Connection header received
0620 18:41:42.068702 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:42.068706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:42.068771 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:42.068882 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:42.070277 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:42.071008 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:42.072265 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:42.072734 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:42.072872 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:42.077541 (+ 4669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:42.077546 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:42.077549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:42.077584 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:42.080481 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:42.081115 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:42.081121 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:42.081127 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:42.081254 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:42.081663 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:42.081669 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:42.081673 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:42.082090 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:42.082427 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:42.082881 (+ 454us) server_negotiation.cc:299] Negotiation successful
0620 18:41:42.083235 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 18:41:42 dist-test-slave-k6lg krb5kdc[6552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 18:41:42 dist-test-slave-k6lg krb5kdc[6552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.129@KRBTEST.COM
I20260620 18:41:42.128754 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:42.114427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53518 (local address 127.5.225.129:43997)
0620 18:41:42.114727 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:42.114733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:42.114934 (+ 201us) server_negotiation.cc:407] Connection header received
0620 18:41:42.115128 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:42.115133 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:42.115217 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:42.115328 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:42.116647 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:42.117507 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:42.118813 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:42.119217 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:42.119314 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:42.123133 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:42.123139 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:42.123143 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:42.123179 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:42.126016 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:42.126760 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:42.126765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:42.126768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:42.126906 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:42.127344 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:42.127348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:42.127350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:42.127687 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:42.127879 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:42.128164 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 18:41:42.128335 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
W20260620 18:41:42.130566 6701 tablet_service.cc:2908] Rejecting scan request for tablet 718e9b57b4514963ab272ee0540ae8b8: Uninitialized: clean time has not yet been initialized
I20260620 18:41:42.133476 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 6658
I20260620 18:41:42.146847 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:42.134449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53530 (local address 127.5.225.129:43997)
0620 18:41:42.134704 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:42.134709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:42.134915 (+ 206us) server_negotiation.cc:407] Connection header received
0620 18:41:42.135141 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:42.135145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:42.135214 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:42.135318 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:42.136453 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:42.137192 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:42.138507 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:42.138920 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:42.139055 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:42.141347 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:42.141352 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:42.141355 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:42.141386 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:42.144113 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:42.144802 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:42.144809 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:42.144812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:42.144948 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:42.145319 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:42.145325 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:42.145328 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:42.145596 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:42.145871 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:42.146201 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 18:41:42.146463 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 18:41:42.147553 6781 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:42.246886 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 6658
I20260620 18:41:42.263695 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 6792
Jun 20 18:41:42 dist-test-slave-k6lg krb5kdc[6552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.130@KRBTEST.COM
I20260620 18:41:42.278766 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:42.264462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48498 (local address 127.5.225.130:35693)
0620 18:41:42.264657 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:42.264662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:42.264680 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:41:42.264832 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:42.264837 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:42.264906 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:42.265024 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:42.266242 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:42.267027 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:42.268405 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:42.268802 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:42.268943 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:42.272921 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:42.272935 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:42.272949 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:42.272989 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:42.275828 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:42.276611 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:42.276620 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:42.276622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:42.276760 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:42.277128 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:42.277134 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:42.277138 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:42.277519 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:42.277819 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:42.278130 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 18:41:42.278385 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 18:41:42.279500 6915 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:42.366668 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 6792
I20260620 18:41:42.383590 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 6925
Jun 20 18:41:42 dist-test-slave-k6lg krb5kdc[6552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.131@KRBTEST.COM
I20260620 18:41:42.398635 7067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:42.384360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43530 (local address 127.5.225.131:40417)
0620 18:41:42.384613 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:42.384619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:42.384640 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:41:42.384816 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:42.384826 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:42.384892 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:42.385000 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:42.386221 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:42.386939 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:42.388235 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:42.388623 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:42.388754 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:42.392789 (+ 4035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:42.392803 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:42.392820 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:42.392859 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:42.395905 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:42.396715 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:42.396723 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:42.396726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:42.396880 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:42.397275 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:42.397279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:42.397281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:42.397643 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:42.397863 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:42.398091 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 18:41:42.398354 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:41:42.399305 7048 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:42.486105 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 6925
I20260620 18:41:42.502095 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 6565
I20260620 18:41:42.503350 6627 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:42.611989 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 6565
2026-06-20T18:41:42Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4241 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:41:42 dist-test-slave-k6lg krb5kdc[7101](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:41:42 dist-test-slave-k6lg krb5kdc[7101](info): set up 2 sockets
Jun 20 18:41:42 dist-test-slave-k6lg krb5kdc[7101](info): commencing operation
krb5kdc: starting...
W20260620 18:41:43.750259 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.005s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:41:43 dist-test-slave-k6lg krb5kdc[7101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:41:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:41:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
I20260620 18:41:43.929975 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:44115
--webserver_interface=127.5.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:44561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:44115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:44.243247 7114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:44.243513 7114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:44.243693 7114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:44.250690 7114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:41:44.250783 7114 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 18:41:44.250864 7114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:44.250980 7114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:44.251046 7114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:41:44.251155 7114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:41:44.260752 7114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:44561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:44115
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:44115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:41:44.261981 7114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:44.263283 7114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:44.275419 7120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:44.275439 7122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:44.275666 7119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:44.276233 7114 server_base.cc:1034] running on GCE node
I20260620 18:41:44.277057 7114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:44.278472 7114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:44.279721 7114 hybrid_clock.cc:648] HybridClock initialized: now 1781980904279640 us; error 49 us; skew 500 ppm
I20260620 18:41:44.280014 7114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:44 dist-test-slave-k6lg krb5kdc[7101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980904, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:44.283229 7114 init.cc:377] Logged in from keytab as kudu/127.5.225.190@KRBTEST.COM (short username kudu)
I20260620 18:41:44.284744 7114 webserver.cc:460] Webserver started at http://127.5.225.190:33785/ using document root <none> and password file <none>
I20260620 18:41:44.285459 7114 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:44.285545 7114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:44.285815 7114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:44.288731 7114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "f63f8e6e7e064537a0e37780fdbab743"
format_stamp: "Formatted at 2026-06-20 18:41:44 on dist-test-slave-k6lg"
server_key: "3162a75c79f8d68c89657bd30bd97bb3"
server_key_iv: "c150ffadf387c9aa0f8f6ea2c08e2e72"
server_key_version: "encryptionkey@0"
I20260620 18:41:44.289471 7114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "f63f8e6e7e064537a0e37780fdbab743"
format_stamp: "Formatted at 2026-06-20 18:41:44 on dist-test-slave-k6lg"
server_key: "3162a75c79f8d68c89657bd30bd97bb3"
server_key_iv: "c150ffadf387c9aa0f8f6ea2c08e2e72"
server_key_version: "encryptionkey@0"
I20260620 18:41:44.294413 7114 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260620 18:41:44.297685 7129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:44.299038 7114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:41:44.299222 7114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "f63f8e6e7e064537a0e37780fdbab743"
format_stamp: "Formatted at 2026-06-20 18:41:44 on dist-test-slave-k6lg"
server_key: "3162a75c79f8d68c89657bd30bd97bb3"
server_key_iv: "c150ffadf387c9aa0f8f6ea2c08e2e72"
server_key_version: "encryptionkey@0"
I20260620 18:41:44.299445 7114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:44.317469 7114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:44.321501 7114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:44.321790 7114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:44.343662 7114 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:44115
I20260620 18:41:44.343686 7190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:44115 every 8 connection(s)
I20260620 18:41:44.345006 7114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:41:44.346342 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 7114
I20260620 18:41:44.346566 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:41:44.347391 6022 external_mini_cluster.cc:1428] Setting key 1b488d7653d2fca6a34f51f921f35199
I20260620 18:41:44.350033 7191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 18:41:44 dist-test-slave-k6lg krb5kdc[7101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:41:44.363494 7191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f63f8e6e7e064537a0e37780fdbab743: Bootstrap starting.
I20260620 18:41:44.366807 7191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f63f8e6e7e064537a0e37780fdbab743: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:44.367813 7191 log.cc:824] T 00000000000000000000000000000000 P f63f8e6e7e064537a0e37780fdbab743: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:44.369848 7194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:44.349495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39292 (local address 127.5.225.190:44115)
0620 18:41:44.350313 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:44.350325 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:44.350367 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:41:44.351922 (+ 1555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:44.351943 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:44.352317 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:44.352665 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:44.354395 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.355724 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.357457 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.358101 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.358295 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:44.362916 (+ 4621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:44.362924 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:44.362938 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:44.362976 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:44.365936 (+ 2960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:44.366651 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:44.366663 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:44.366669 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:44.366833 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:44.367234 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:44.367243 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:44.367246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:44.367728 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:44.368035 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:44.368488 (+ 453us) server_negotiation.cc:299] Negotiation successful
0620 18:41:44.368885 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":363,"threads_started":1}
I20260620 18:41:44.370784 7191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f63f8e6e7e064537a0e37780fdbab743: No bootstrap required, opened a new log
I20260620 18:41:44.378005 7191 raft_consensus.cc:348] T 00000000000000000000000000000000 P f63f8e6e7e064537a0e37780fdbab743 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f63f8e6e7e064537a0e37780fdbab743" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 44115 } }
I20260620 18:41:44.378459 7191 raft_consensus.cc:374] T 00000000000000000000000000000000 P f63f8e6e7e064537a0e37780fdbab743 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:44.378593 7191 raft_consensus.cc:729] T 00000000000000000000000000000000 P f63f8e6e7e064537a0e37780fdbab743 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f63f8e6e7e064537a0e37780fdbab743, State: Initialized, Role: FOLLOWER
I20260620 18:41:44.379192 7191 consensus_queue.cc:260] T 00000000000000000000000000000000 P f63f8e6e7e064537a0e37780fdbab743 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f63f8e6e7e064537a0e37780fdbab743" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 44115 } }
I20260620 18:41:44.379395 7191 raft_consensus.cc:388] T 00000000000000000000000000000000 P f63f8e6e7e064537a0e37780fdbab743 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:41:44.379549 7191 raft_consensus.cc:482] T 00000000000000000000000000000000 P f63f8e6e7e064537a0e37780fdbab743 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:41:44.379716 7191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f63f8e6e7e064537a0e37780fdbab743 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:44.381799 7191 raft_consensus.cc:504] T 00000000000000000000000000000000 P f63f8e6e7e064537a0e37780fdbab743 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f63f8e6e7e064537a0e37780fdbab743" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 44115 } }
I20260620 18:41:44.382396 7191 leader_election.cc:302] T 00000000000000000000000000000000 P f63f8e6e7e064537a0e37780fdbab743 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f63f8e6e7e064537a0e37780fdbab743; no voters:
I20260620 18:41:44.383111 7191 leader_election.cc:288] T 00000000000000000000000000000000 P f63f8e6e7e064537a0e37780fdbab743 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:41:44.383232 7196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f63f8e6e7e064537a0e37780fdbab743 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:41:44.385138 7196 raft_consensus.cc:686] T 00000000000000000000000000000000 P f63f8e6e7e064537a0e37780fdbab743 [term 1 LEADER]: Becoming Leader. State: Replica: f63f8e6e7e064537a0e37780fdbab743, State: Running, Role: LEADER
I20260620 18:41:44.385494 7191 sys_catalog.cc:564] T 00000000000000000000000000000000 P f63f8e6e7e064537a0e37780fdbab743 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:41:44.385725 7196 consensus_queue.cc:237] T 00000000000000000000000000000000 P f63f8e6e7e064537a0e37780fdbab743 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f63f8e6e7e064537a0e37780fdbab743" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 44115 } }
I20260620 18:41:44.388693 7198 sys_catalog.cc:455] T 00000000000000000000000000000000 P f63f8e6e7e064537a0e37780fdbab743 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f63f8e6e7e064537a0e37780fdbab743. Latest consensus state: current_term: 1 leader_uuid: "f63f8e6e7e064537a0e37780fdbab743" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f63f8e6e7e064537a0e37780fdbab743" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 44115 } } }
I20260620 18:41:44.388667 7197 sys_catalog.cc:455] T 00000000000000000000000000000000 P f63f8e6e7e064537a0e37780fdbab743 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f63f8e6e7e064537a0e37780fdbab743" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f63f8e6e7e064537a0e37780fdbab743" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 44115 } } }
I20260620 18:41:44.389103 7198 sys_catalog.cc:458] T 00000000000000000000000000000000 P f63f8e6e7e064537a0e37780fdbab743 [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:44.389107 7197 sys_catalog.cc:458] T 00000000000000000000000000000000 P f63f8e6e7e064537a0e37780fdbab743 [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:44.391716 7204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:41:44.396809 7204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:41:44.405378 7204 catalog_manager.cc:1347] Generated new cluster ID: 22b3632eff744089871027a68536c278
I20260620 18:41:44.405484 7204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:41:44.437028 7204 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:41:44.438459 7204 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:41:44.450997 7204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f63f8e6e7e064537a0e37780fdbab743: Generated new TSK 0
I20260620 18:41:44.451882 7204 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab.
Entry for principal kudu/127.5.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab.
I20260620 18:41:44.501633 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.129:0
--local_ip_for_outbound_sockets=127.5.225.129
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:44115
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:44561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:44.831103 7217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:44.831383 7217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:44.831487 7217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:44.838555 7217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:44.838663 7217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:44.838747 7217 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:41:44.838892 7217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.129
I20260620 18:41:44.849633 7217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:44561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.225.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.225.190:44115
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.129
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:41:44.851042 7217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:44.852355 7217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:44.865360 7222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:44.865473 7223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:44.865674 7225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:44.866466 7217 server_base.cc:1034] running on GCE node
I20260620 18:41:44.867136 7217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:44.868091 7217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:44.869365 7217 hybrid_clock.cc:648] HybridClock initialized: now 1781980904869279 us; error 36 us; skew 500 ppm
I20260620 18:41:44.869719 7217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:44 dist-test-slave-k6lg krb5kdc[7101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980904, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:44.873014 7217 init.cc:377] Logged in from keytab as kudu/127.5.225.129@KRBTEST.COM (short username kudu)
I20260620 18:41:44.874652 7217 webserver.cc:460] Webserver started at http://127.5.225.129:44837/ using document root <none> and password file <none>
I20260620 18:41:44.875347 7217 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:44.875481 7217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:44.875798 7217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:44.878638 7217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-0/data/instance:
uuid: "2d027e7df09047df8520a2776a891718"
format_stamp: "Formatted at 2026-06-20 18:41:44 on dist-test-slave-k6lg"
server_key: "36d7f079a93c04ee7b83c47ac046d4cf"
server_key_iv: "d24a3af9675dd37db88d77cb4142d941"
server_key_version: "encryptionkey@0"
I20260620 18:41:44.879740 7217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance:
uuid: "2d027e7df09047df8520a2776a891718"
format_stamp: "Formatted at 2026-06-20 18:41:44 on dist-test-slave-k6lg"
server_key: "36d7f079a93c04ee7b83c47ac046d4cf"
server_key_iv: "d24a3af9675dd37db88d77cb4142d941"
server_key_version: "encryptionkey@0"
I20260620 18:41:44.884706 7217 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:41:44.888171 7232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:44.889422 7217 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 18:41:44.889585 7217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-0/wal
uuid: "2d027e7df09047df8520a2776a891718"
format_stamp: "Formatted at 2026-06-20 18:41:44 on dist-test-slave-k6lg"
server_key: "36d7f079a93c04ee7b83c47ac046d4cf"
server_key_iv: "d24a3af9675dd37db88d77cb4142d941"
server_key_version: "encryptionkey@0"
I20260620 18:41:44.889791 7217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:44.901484 7217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:44.905795 7217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:44.906066 7217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:44.908519 7217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:44.908630 7217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:44.908757 7217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:44.908833 7217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:41:44 dist-test-slave-k6lg krb5kdc[7101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980904, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:41:44.933496 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:44.909198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:38009 (local address 127.5.225.190:44115)
0620 18:41:44.910437 (+ 1239us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:44.910445 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:44.910998 (+ 553us) server_negotiation.cc:407] Connection header received
0620 18:41:44.912981 (+ 1983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:44.912987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:44.913089 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:44.913208 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:44.915560 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.916835 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.920142 (+ 3307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.920657 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.920800 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:44.926320 (+ 5520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:44.926326 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:44.926329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:44.926361 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:44.929520 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:44.930610 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:44.930617 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:44.930622 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:44.930774 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:44.931415 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:44.931420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:44.931424 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:44.931767 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:44.931930 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:44.932906 (+ 976us) server_negotiation.cc:299] Negotiation successful
0620 18:41:44.933118 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1099,"thread_start_us":994,"threads_started":1}
I20260620 18:41:44.934815 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:44.909978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:44115 (local address 127.5.225.129:38009)
0620 18:41:44.910844 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:44.910881 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:44.912707 (+ 1826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:44.913513 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:44.913527 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:44.914343 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:44.915291 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:44.915313 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.917054 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.917060 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:44.919944 (+ 2884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:44.919964 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.920951 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.920962 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:44.921311 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:44.922135 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:44.922197 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:44.926027 (+ 3830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:44.929671 (+ 3644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:44.929682 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:44.929702 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:44.930433 (+ 731us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:44.930869 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:44.930878 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:44.930882 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:44.931168 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:44.932013 (+ 845us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:44.932027 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:44.932506 (+ 479us) client_negotiation.cc:769] Sending connection context
0620 18:41:44.932823 (+ 317us) client_negotiation.cc:240] Negotiation successful
0620 18:41:44.933391 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":385,"threads_started":1}
I20260620 18:41:44.956503 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:44.940826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:34037 (local address 127.5.225.190:44115)
0620 18:41:44.941004 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:44.941012 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:44.941816 (+ 804us) server_negotiation.cc:407] Connection header received
0620 18:41:44.942012 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:44.942017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:44.942090 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:44.942232 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:44.943790 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.944490 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.946999 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.947388 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.947527 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:44.950284 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:44.950293 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:44.950301 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:44.950345 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:44.953178 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:44.953817 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:44.953823 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:44.953826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:44.953950 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:44.954388 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:44.954395 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:44.954398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:44.954705 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:44.954862 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:44.955832 (+ 970us) server_negotiation.cc:299] Negotiation successful
0620 18:41:44.956102 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 18:41:44.956545 7316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:44.940893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:44115 (local address 127.5.225.129:34037)
0620 18:41:44.941716 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:44.941734 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:44.941887 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:44.942277 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:44.942285 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:44.943035 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:44.943623 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:44.943636 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.944637 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.944645 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:44.946858 (+ 2213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:44.946874 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:44.947648 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:44.947655 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:44.947913 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:44.948659 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:44.948693 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:44.950078 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:44.953314 (+ 3236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:44.953322 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:44.953327 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:44.953697 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:44.954078 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:44.954084 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:44.954088 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:44.954280 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:44.954843 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:44.954851 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:44.955118 (+ 267us) client_negotiation.cc:769] Sending connection context
0620 18:41:44.955304 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 18:41:44.955879 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"thread_start_us":581,"threads_started":1}
I20260620 18:41:44.958530 7217 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.129:35637
I20260620 18:41:44.958530 7352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.129:35637 every 8 connection(s)
I20260620 18:41:44.959847 7217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
I20260620 18:41:44.961881 7353 heartbeater.cc:347] Connected to a master server at 127.5.225.190:44115
I20260620 18:41:44.962117 7353 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:44.962198 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 7217
I20260620 18:41:44.962430 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance
I20260620 18:41:44.962817 7353 heartbeater.cc:511] Master 127.5.225.190:44115 requested a full tablet report, sending...
I20260620 18:41:44.963106 6022 external_mini_cluster.cc:1428] Setting key 1cfdda5383162ec451a9ee50ea6cfee5
I20260620 18:41:44.964982 7146 ts_manager.cc:194] Registered new tserver with Master: 2d027e7df09047df8520a2776a891718 (127.5.225.129:35637)
I20260620 18:41:44.967096 7146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.129@KRBTEST.COM'} at 127.5.225.129:38009
WARNING: no policy specified for kudu/127.5.225.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab.
Entry for principal kudu/127.5.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab.
I20260620 18:41:45.001004 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.130:0
--local_ip_for_outbound_sockets=127.5.225.130
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:44115
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:44561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:45.316818 7357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:45.317152 7357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:45.317324 7357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:45.324594 7357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:45.324762 7357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:45.324914 7357 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:41:45.325057 7357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.130
I20260620 18:41:45.334606 7357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:44561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.225.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.225.190:44115
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.130
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:41:45.335906 7357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:45.337282 7357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:45.350654 7362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:45.350625 7363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:45.350993 7365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:45.351961 7357 server_base.cc:1034] running on GCE node
I20260620 18:41:45.352586 7357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:45.353579 7357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:45.354813 7357 hybrid_clock.cc:648] HybridClock initialized: now 1781980905354748 us; error 43 us; skew 500 ppm
I20260620 18:41:45.355103 7357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:45 dist-test-slave-k6lg krb5kdc[7101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980905, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:45.358224 7357 init.cc:377] Logged in from keytab as kudu/127.5.225.130@KRBTEST.COM (short username kudu)
I20260620 18:41:45.359691 7357 webserver.cc:460] Webserver started at http://127.5.225.130:35295/ using document root <none> and password file <none>
I20260620 18:41:45.360373 7357 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:45.360456 7357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:45.360760 7357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:45.363622 7357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-1/data/instance:
uuid: "cd6a34c3397b44ce85248a5a9c85f066"
format_stamp: "Formatted at 2026-06-20 18:41:45 on dist-test-slave-k6lg"
server_key: "020fac3772a79569b7319dc490395052"
server_key_iv: "9dcd0f2f3ff3a7ceeb2ed0d7c66676ad"
server_key_version: "encryptionkey@0"
I20260620 18:41:45.364394 7357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance:
uuid: "cd6a34c3397b44ce85248a5a9c85f066"
format_stamp: "Formatted at 2026-06-20 18:41:45 on dist-test-slave-k6lg"
server_key: "020fac3772a79569b7319dc490395052"
server_key_iv: "9dcd0f2f3ff3a7ceeb2ed0d7c66676ad"
server_key_version: "encryptionkey@0"
I20260620 18:41:45.369318 7357 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260620 18:41:45.372898 7372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:45.374302 7357 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:41:45.374454 7357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-1/wal
uuid: "cd6a34c3397b44ce85248a5a9c85f066"
format_stamp: "Formatted at 2026-06-20 18:41:45 on dist-test-slave-k6lg"
server_key: "020fac3772a79569b7319dc490395052"
server_key_iv: "9dcd0f2f3ff3a7ceeb2ed0d7c66676ad"
server_key_version: "encryptionkey@0"
I20260620 18:41:45.374652 7357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:45.396754 7357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:45.402266 7357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:45.402554 7357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:45.405354 7357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:45.405459 7357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:45.405599 7357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:45.405686 7357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:41:45 dist-test-slave-k6lg krb5kdc[7101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980905, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:41:45.426227 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:45.405826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:34315 (local address 127.5.225.190:44115)
0620 18:41:45.406004 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:45.406010 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:45.407750 (+ 1740us) server_negotiation.cc:407] Connection header received
0620 18:41:45.409945 (+ 2195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:45.409951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:45.410047 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:45.410156 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:45.411608 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:45.412342 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:45.413922 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:45.414393 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:45.414526 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:45.419567 (+ 5041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:45.419577 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:45.419584 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:45.419634 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:45.422929 (+ 3295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:45.423880 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:45.423888 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:45.423892 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:45.424042 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:45.424464 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:45.424468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:45.424472 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:45.424768 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:45.424936 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:45.425612 (+ 676us) server_negotiation.cc:299] Negotiation successful
0620 18:41:45.425836 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:41:45.427491 7381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:45.406657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:44115 (local address 127.5.225.130:34315)
0620 18:41:45.407589 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:45.407630 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:45.409646 (+ 2016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:45.410416 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:45.410423 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:45.410902 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:45.411439 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:45.411455 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:45.412487 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:45.412503 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:45.413783 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:45.413795 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:45.414650 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:45.414659 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:45.414869 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:45.415701 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:45.415731 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:45.419315 (+ 3584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:45.423091 (+ 3776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:45.423097 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:45.423110 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:45.423749 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:45.424166 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:45.424169 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:45.424172 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:45.424347 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:45.424903 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:45.424910 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:45.425261 (+ 351us) client_negotiation.cc:769] Sending connection context
0620 18:41:45.425532 (+ 271us) client_negotiation.cc:240] Negotiation successful
0620 18:41:45.426039 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":392,"threads_started":1}
I20260620 18:41:45.453135 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:45.437911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:37479 (local address 127.5.225.190:44115)
0620 18:41:45.438129 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:45.438138 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:45.438767 (+ 629us) server_negotiation.cc:407] Connection header received
0620 18:41:45.438961 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:45.438965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:45.439041 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:45.439120 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:45.440517 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:45.441211 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:45.443694 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:45.444319 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:45.444463 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:45.447166 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:45.447171 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:45.447174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:45.447203 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:45.450104 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:45.451141 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:45.451146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:45.451149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:45.451308 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:45.451783 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:45.451787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:45.451789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:45.452105 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:45.452216 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:45.452754 (+ 538us) server_negotiation.cc:299] Negotiation successful
0620 18:41:45.452916 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 18:41:45.453758 7431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:45.437857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:44115 (local address 127.5.225.130:37479)
0620 18:41:45.438640 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:45.438666 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:45.438839 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:45.439185 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:45.439193 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:45.439892 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:45.440356 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:45.440372 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:45.441346 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:45.441354 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:45.443537 (+ 2183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:45.443555 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:45.444537 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:45.444550 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:45.444837 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:45.445691 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:45.445712 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:45.446976 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:45.450281 (+ 3305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:45.450286 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:45.450290 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:45.450986 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:45.451450 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:45.451457 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:45.451464 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:45.451657 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:45.452221 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:45.452228 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:45.452491 (+ 263us) client_negotiation.cc:769] Sending connection context
0620 18:41:45.452672 (+ 181us) client_negotiation.cc:240] Negotiation successful
0620 18:41:45.453111 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":394,"threads_started":1}
I20260620 18:41:45.469005 7357 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.130:33533
I20260620 18:41:45.469053 7491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.130:33533 every 8 connection(s)
I20260620 18:41:45.470288 7357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
I20260620 18:41:45.472716 7492 heartbeater.cc:347] Connected to a master server at 127.5.225.190:44115
I20260620 18:41:45.472923 7492 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:45.473502 7492 heartbeater.cc:511] Master 127.5.225.190:44115 requested a full tablet report, sending...
I20260620 18:41:45.474800 7146 ts_manager.cc:194] Registered new tserver with Master: cd6a34c3397b44ce85248a5a9c85f066 (127.5.225.130:33533)
I20260620 18:41:45.475556 7146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.130@KRBTEST.COM'} at 127.5.225.130:34315
I20260620 18:41:45.478890 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 7357
I20260620 18:41:45.479058 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance
I20260620 18:41:45.479496 6022 external_mini_cluster.cc:1428] Setting key 2825861d588dbf439d1bb7eeba137a78
WARNING: no policy specified for kudu/127.5.225.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab.
Entry for principal kudu/127.5.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab.
I20260620 18:41:45.519461 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.131:0
--local_ip_for_outbound_sockets=127.5.225.131
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:44115
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:44561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:45.837595 7496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:45.837874 7496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:45.838055 7496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:45.845345 7496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:45.845463 7496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:45.845621 7496 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:41:45.845739 7496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.131
I20260620 18:41:45.855959 7496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:44561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.225.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.225.190:44115
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.131
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:41:45.857335 7496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:45.858777 7496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:45.872105 7504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:45.872373 7502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:45.872138 7501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:45.873023 7496 server_base.cc:1034] running on GCE node
I20260620 18:41:45.873704 7496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:45.874753 7496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:45.875999 7496 hybrid_clock.cc:648] HybridClock initialized: now 1781980905875926 us; error 55 us; skew 500 ppm
I20260620 18:41:45.876314 7496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:45 dist-test-slave-k6lg krb5kdc[7101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980905, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:45.879580 7496 init.cc:377] Logged in from keytab as kudu/127.5.225.131@KRBTEST.COM (short username kudu)
I20260620 18:41:45.881080 7496 webserver.cc:460] Webserver started at http://127.5.225.131:44685/ using document root <none> and password file <none>
I20260620 18:41:45.881781 7496 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:45.881892 7496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:45.882288 7496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:45.885174 7496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-2/data/instance:
uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd"
format_stamp: "Formatted at 2026-06-20 18:41:45 on dist-test-slave-k6lg"
server_key: "7de2a75952266bb31288234468428066"
server_key_iv: "3e2bb4942ff9cb442f5376912a0f15c0"
server_key_version: "encryptionkey@0"
I20260620 18:41:45.886008 7496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance:
uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd"
format_stamp: "Formatted at 2026-06-20 18:41:45 on dist-test-slave-k6lg"
server_key: "7de2a75952266bb31288234468428066"
server_key_iv: "3e2bb4942ff9cb442f5376912a0f15c0"
server_key_version: "encryptionkey@0"
I20260620 18:41:45.891574 7496 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:41:45.895484 7511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:45.896908 7496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:41:45.897091 7496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-2/wal
uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd"
format_stamp: "Formatted at 2026-06-20 18:41:45 on dist-test-slave-k6lg"
server_key: "7de2a75952266bb31288234468428066"
server_key_iv: "3e2bb4942ff9cb442f5376912a0f15c0"
server_key_version: "encryptionkey@0"
I20260620 18:41:45.897313 7496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:45.916461 7496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:45.920361 7496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:45.920639 7496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:45.923383 7496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:45.923491 7496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:45.923696 7496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:45.923768 7496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:41:45 dist-test-slave-k6lg krb5kdc[7101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980905, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:41:45.946712 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:45.923863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:40669 (local address 127.5.225.190:44115)
0620 18:41:45.924098 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:45.924106 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:45.925483 (+ 1377us) server_negotiation.cc:407] Connection header received
0620 18:41:45.927652 (+ 2169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:45.927657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:45.927769 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:45.927879 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:45.930301 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:45.931653 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:45.934531 (+ 2878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:45.934946 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:45.935090 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:45.940051 (+ 4961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:45.940056 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:45.940059 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:45.940091 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:45.942954 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:45.943855 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:45.943861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:45.943866 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:45.944011 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:45.944473 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:45.944477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:45.944479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:45.944749 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:45.944908 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:45.945852 (+ 944us) server_negotiation.cc:299] Negotiation successful
0620 18:41:45.946198 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 18:41:45.947543 7520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:45.924500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:44115 (local address 127.5.225.131:40669)
0620 18:41:45.925307 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:45.925351 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:45.927353 (+ 2002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:45.928125 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:45.928136 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:45.929010 (+ 874us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:45.929998 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:45.930022 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:45.931918 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:45.931924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:45.934346 (+ 2422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:45.934370 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:45.935192 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:45.935203 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:45.935556 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:45.936473 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:45.936499 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:45.939807 (+ 3308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:45.943104 (+ 3297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:45.943113 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:45.943131 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:45.943601 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:45.944126 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:45.944132 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:45.944136 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:45.944328 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:45.944878 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:45.944886 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:45.945302 (+ 416us) client_negotiation.cc:769] Sending connection context
0620 18:41:45.945628 (+ 326us) client_negotiation.cc:240] Negotiation successful
0620 18:41:45.946025 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":370,"threads_started":1}
I20260620 18:41:45.969822 7353 heartbeater.cc:503] Master 127.5.225.190:44115 was elected leader, sending a full tablet report...
I20260620 18:41:45.973613 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:45.956330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:49137 (local address 127.5.225.190:44115)
0620 18:41:45.956531 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:45.956538 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:45.957545 (+ 1007us) server_negotiation.cc:407] Connection header received
0620 18:41:45.957787 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:45.957793 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:45.957872 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:45.957989 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:45.959132 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:45.960681 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:45.962814 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:45.963661 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:45.963827 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:45.965754 (+ 1927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:45.965760 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:45.965764 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:45.965806 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:45.970000 (+ 4194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:45.971031 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:45.971037 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:45.971041 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:45.971225 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:45.971718 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:45.971725 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:45.971728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:45.972146 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:45.972313 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:45.972903 (+ 590us) server_negotiation.cc:299] Negotiation successful
0620 18:41:45.973188 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:41:45.973989 7577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:45.956404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:44115 (local address 127.5.225.131:49137)
0620 18:41:45.957382 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:45.957406 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:45.957627 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:45.958009 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:45.958013 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:45.958575 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:45.958952 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:45.958969 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:45.960938 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:45.960943 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:45.962647 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:45.962662 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:45.963895 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:45.963903 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:45.964083 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:45.964817 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:45.964833 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:45.965533 (+ 700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:45.970147 (+ 4614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:45.970155 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:45.970182 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:45.970841 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:45.971374 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:45.971381 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:45.971387 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:45.971591 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:45.972292 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:45.972304 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:45.972570 (+ 266us) client_negotiation.cc:769] Sending connection context
0620 18:41:45.972760 (+ 190us) client_negotiation.cc:240] Negotiation successful
0620 18:41:45.973271 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"thread_start_us":684,"threads_started":1}
I20260620 18:41:45.982775 7496 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.131:46683
I20260620 18:41:45.982798 7630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.131:46683 every 8 connection(s)
I20260620 18:41:45.983932 7496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
I20260620 18:41:45.985877 7631 heartbeater.cc:347] Connected to a master server at 127.5.225.190:44115
I20260620 18:41:45.986104 7631 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:45.986953 7631 heartbeater.cc:511] Master 127.5.225.190:44115 requested a full tablet report, sending...
I20260620 18:41:45.987172 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 7496
I20260620 18:41:45.987370 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance
I20260620 18:41:45.987785 6022 external_mini_cluster.cc:1428] Setting key 57c88d73780c419938a2096e4268aa4c
I20260620 18:41:45.988482 7146 ts_manager.cc:194] Registered new tserver with Master: c20fc465cf9f4ea7b5cddf62855dc2bd (127.5.225.131:46683)
I20260620 18:41:45.989614 7146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.131@KRBTEST.COM'} at 127.5.225.131:40669
I20260620 18:41:45.990867 6022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 18:41:46 dist-test-slave-k6lg krb5kdc[7101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980906, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 18:41:46 dist-test-slave-k6lg krb5kdc[7101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980906, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:41:46.035982 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:46.022055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39296 (local address 127.5.225.190:44115)
0620 18:41:46.022299 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:46.022308 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:46.022480 (+ 172us) server_negotiation.cc:407] Connection header received
0620 18:41:46.022641 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:46.022645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:46.022710 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:46.022795 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:46.024135 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.024884 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.026124 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.026602 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.026701 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:46.030664 (+ 3963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:46.030669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:46.030672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:46.030706 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:46.033374 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:46.034010 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:46.034014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:46.034016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:46.034127 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:46.034591 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:46.034595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:46.034597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:46.034886 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:46.035086 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:46.035359 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 18:41:46.035530 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 18:41:46.040127 7145 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:41:46.046864 7145 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:41:46.073448 7644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:46.059326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:35637 (local address 127.0.0.1:58966)
0620 18:41:46.060161 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:46.060201 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:46.060364 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:46.061616 (+ 1252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:46.061627 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:46.061668 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:46.062269 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:46.062287 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.066374 (+ 4087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.066384 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:46.069582 (+ 3198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:46.069605 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.072279 (+ 2674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.072289 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:46.072604 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:46.072684 (+ 80us) client_negotiation.cc:769] Sending connection context
0620 18:41:46.072836 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 18:41:46.073059 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":335,"threads_started":1}
I20260620 18:41:46.074563 7645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:46.059503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58966 (local address 127.5.225.129:35637)
0620 18:41:46.060886 (+ 1383us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:46.060894 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:46.060923 (+ 29us) server_negotiation.cc:407] Connection header received
0620 18:41:46.061025 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:46.061033 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:46.061333 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:46.062536 (+ 1203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:46.062783 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.066154 (+ 3371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.069855 (+ 3701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.071983 (+ 2128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.073731 (+ 1748us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:46.073962 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:46.074028 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 18:41:46.074242 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1233,"thread_start_us":1456,"threads_started":1}
I20260620 18:41:46.080530 7283 tablet_service.cc:1467] Processing CreateTablet for tablet 17ee9881ee224beb8363944baf158721 (DEFAULT_TABLE table=test-table [id=ba46a03d065b491eb2fbf8a1a5510a14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:41:46.081856 7283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17ee9881ee224beb8363944baf158721. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:46.082854 7644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:46.059781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.130:33533 (local address 127.0.0.1:51856)
0620 18:41:46.073706 (+ 13925us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:46.073725 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:46.073819 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:46.075871 (+ 2052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:46.075876 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:46.075905 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:46.076040 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:46.076048 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.078114 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.078119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:46.080179 (+ 2060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:46.080193 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.081982 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.081988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:46.082179 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:46.082215 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 18:41:46.082318 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 18:41:46.082460 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13814,"mutex_wait_us":484,"thread_start_us":13997,"threads_started":1}
I20260620 18:41:46.083448 7646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:46.060120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51856 (local address 127.5.225.130:33533)
0620 18:41:46.075292 (+ 15172us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:46.075299 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:46.075323 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:41:46.075405 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:46.075411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:46.075643 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:46.076244 (+ 601us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:46.076449 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.077953 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.080420 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.081806 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.081940 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:46.082140 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:46.082376 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 18:41:46.082553 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15030,"thread_start_us":14912,"threads_started":1}
I20260620 18:41:46.083879 7649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:46.060834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59066 (local address 127.5.225.131:46683)
0620 18:41:46.073329 (+ 12495us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:46.073338 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:46.074995 (+ 1657us) server_negotiation.cc:407] Connection header received
0620 18:41:46.075131 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:46.075138 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:46.075403 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:46.075590 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:46.076380 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.077964 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.080326 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.081806 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.081939 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:46.082206 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:46.083369 (+ 1163us) server_negotiation.cc:299] Negotiation successful
0620 18:41:46.083557 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12307,"thread_start_us":12513,"threads_started":1}
I20260620 18:41:46.084482 7647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:46.060469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.131:46683 (local address 127.0.0.1:59066)
0620 18:41:46.074592 (+ 14123us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:46.074611 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:46.074848 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:46.075670 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:46.075674 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:46.075706 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:46.076173 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:46.076185 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.078149 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.078153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:46.080167 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:46.080180 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.082874 (+ 2694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.082881 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:46.083041 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:46.083083 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 18:41:46.083169 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 18:41:46.084137 (+ 968us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13979,"mutex_wait_us":8,"thread_start_us":14978,"threads_started":1}
I20260620 18:41:46.087198 7425 tablet_service.cc:1467] Processing CreateTablet for tablet 17ee9881ee224beb8363944baf158721 (DEFAULT_TABLE table=test-table [id=ba46a03d065b491eb2fbf8a1a5510a14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:41:46.087922 7563 tablet_service.cc:1467] Processing CreateTablet for tablet 17ee9881ee224beb8363944baf158721 (DEFAULT_TABLE table=test-table [id=ba46a03d065b491eb2fbf8a1a5510a14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:41:46.088702 7425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17ee9881ee224beb8363944baf158721. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:46.089092 7563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17ee9881ee224beb8363944baf158721. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:46.099663 7650 tablet_bootstrap.cc:492] T 17ee9881ee224beb8363944baf158721 P 2d027e7df09047df8520a2776a891718: Bootstrap starting.
I20260620 18:41:46.102934 7651 tablet_bootstrap.cc:492] T 17ee9881ee224beb8363944baf158721 P c20fc465cf9f4ea7b5cddf62855dc2bd: Bootstrap starting.
I20260620 18:41:46.104319 7652 tablet_bootstrap.cc:492] T 17ee9881ee224beb8363944baf158721 P cd6a34c3397b44ce85248a5a9c85f066: Bootstrap starting.
I20260620 18:41:46.104506 7650 tablet_bootstrap.cc:654] T 17ee9881ee224beb8363944baf158721 P 2d027e7df09047df8520a2776a891718: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:46.106302 7650 log.cc:824] T 17ee9881ee224beb8363944baf158721 P 2d027e7df09047df8520a2776a891718: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:46.106603 7651 tablet_bootstrap.cc:654] T 17ee9881ee224beb8363944baf158721 P c20fc465cf9f4ea7b5cddf62855dc2bd: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:46.107456 7652 tablet_bootstrap.cc:654] T 17ee9881ee224beb8363944baf158721 P cd6a34c3397b44ce85248a5a9c85f066: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:46.107887 7651 log.cc:824] T 17ee9881ee224beb8363944baf158721 P c20fc465cf9f4ea7b5cddf62855dc2bd: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:46.108660 7652 log.cc:824] T 17ee9881ee224beb8363944baf158721 P cd6a34c3397b44ce85248a5a9c85f066: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:46.109732 7650 tablet_bootstrap.cc:492] T 17ee9881ee224beb8363944baf158721 P 2d027e7df09047df8520a2776a891718: No bootstrap required, opened a new log
I20260620 18:41:46.110095 7650 ts_tablet_manager.cc:1397] T 17ee9881ee224beb8363944baf158721 P 2d027e7df09047df8520a2776a891718: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260620 18:41:46.110460 7651 tablet_bootstrap.cc:492] T 17ee9881ee224beb8363944baf158721 P c20fc465cf9f4ea7b5cddf62855dc2bd: No bootstrap required, opened a new log
I20260620 18:41:46.110713 7651 ts_tablet_manager.cc:1397] T 17ee9881ee224beb8363944baf158721 P c20fc465cf9f4ea7b5cddf62855dc2bd: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 18:41:46.117682 7652 tablet_bootstrap.cc:492] T 17ee9881ee224beb8363944baf158721 P cd6a34c3397b44ce85248a5a9c85f066: No bootstrap required, opened a new log
I20260620 18:41:46.117969 7652 ts_tablet_manager.cc:1397] T 17ee9881ee224beb8363944baf158721 P cd6a34c3397b44ce85248a5a9c85f066: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260620 18:41:46.118324 7651 raft_consensus.cc:348] T 17ee9881ee224beb8363944baf158721 P c20fc465cf9f4ea7b5cddf62855dc2bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d027e7df09047df8520a2776a891718" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35637 } } peers { permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 } } peers { permanent_uuid: "cd6a34c3397b44ce85248a5a9c85f066" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33533 } }
I20260620 18:41:46.118685 7651 raft_consensus.cc:374] T 17ee9881ee224beb8363944baf158721 P c20fc465cf9f4ea7b5cddf62855dc2bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:46.118788 7651 raft_consensus.cc:729] T 17ee9881ee224beb8363944baf158721 P c20fc465cf9f4ea7b5cddf62855dc2bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c20fc465cf9f4ea7b5cddf62855dc2bd, State: Initialized, Role: FOLLOWER
I20260620 18:41:46.119444 7651 consensus_queue.cc:260] T 17ee9881ee224beb8363944baf158721 P c20fc465cf9f4ea7b5cddf62855dc2bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d027e7df09047df8520a2776a891718" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35637 } } peers { permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 } } peers { permanent_uuid: "cd6a34c3397b44ce85248a5a9c85f066" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33533 } }
I20260620 18:41:46.120513 7631 heartbeater.cc:503] Master 127.5.225.190:44115 was elected leader, sending a full tablet report...
I20260620 18:41:46.121206 7651 ts_tablet_manager.cc:1428] T 17ee9881ee224beb8363944baf158721 P c20fc465cf9f4ea7b5cddf62855dc2bd: Time spent starting tablet: real 0.010s user 0.006s sys 0.003s
I20260620 18:41:46.122593 7650 raft_consensus.cc:348] T 17ee9881ee224beb8363944baf158721 P 2d027e7df09047df8520a2776a891718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d027e7df09047df8520a2776a891718" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35637 } } peers { permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 } } peers { permanent_uuid: "cd6a34c3397b44ce85248a5a9c85f066" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33533 } }
I20260620 18:41:46.123023 7650 raft_consensus.cc:374] T 17ee9881ee224beb8363944baf158721 P 2d027e7df09047df8520a2776a891718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:46.123131 7650 raft_consensus.cc:729] T 17ee9881ee224beb8363944baf158721 P 2d027e7df09047df8520a2776a891718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d027e7df09047df8520a2776a891718, State: Initialized, Role: FOLLOWER
I20260620 18:41:46.123751 7650 consensus_queue.cc:260] T 17ee9881ee224beb8363944baf158721 P 2d027e7df09047df8520a2776a891718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d027e7df09047df8520a2776a891718" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35637 } } peers { permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 } } peers { permanent_uuid: "cd6a34c3397b44ce85248a5a9c85f066" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33533 } }
I20260620 18:41:46.130811 7652 raft_consensus.cc:348] T 17ee9881ee224beb8363944baf158721 P cd6a34c3397b44ce85248a5a9c85f066 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d027e7df09047df8520a2776a891718" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35637 } } peers { permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 } } peers { permanent_uuid: "cd6a34c3397b44ce85248a5a9c85f066" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33533 } }
I20260620 18:41:46.131243 7652 raft_consensus.cc:374] T 17ee9881ee224beb8363944baf158721 P cd6a34c3397b44ce85248a5a9c85f066 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:46.131348 7652 raft_consensus.cc:729] T 17ee9881ee224beb8363944baf158721 P cd6a34c3397b44ce85248a5a9c85f066 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd6a34c3397b44ce85248a5a9c85f066, State: Initialized, Role: FOLLOWER
I20260620 18:41:46.131803 7650 ts_tablet_manager.cc:1428] T 17ee9881ee224beb8363944baf158721 P 2d027e7df09047df8520a2776a891718: Time spent starting tablet: real 0.022s user 0.012s sys 0.007s
I20260620 18:41:46.132072 7652 consensus_queue.cc:260] T 17ee9881ee224beb8363944baf158721 P cd6a34c3397b44ce85248a5a9c85f066 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d027e7df09047df8520a2776a891718" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35637 } } peers { permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 } } peers { permanent_uuid: "cd6a34c3397b44ce85248a5a9c85f066" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33533 } }
I20260620 18:41:46.133807 7652 ts_tablet_manager.cc:1428] T 17ee9881ee224beb8363944baf158721 P cd6a34c3397b44ce85248a5a9c85f066: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260620 18:41:46.134035 7492 heartbeater.cc:503] Master 127.5.225.190:44115 was elected leader, sending a full tablet report...
W20260620 18:41:46.212416 7354 tablet.cc:2369] T 17ee9881ee224beb8363944baf158721 P 2d027e7df09047df8520a2776a891718: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:41:46.222788 7493 tablet.cc:2369] T 17ee9881ee224beb8363944baf158721 P cd6a34c3397b44ce85248a5a9c85f066: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:41:46.223392 7657 raft_consensus.cc:482] T 17ee9881ee224beb8363944baf158721 P 2d027e7df09047df8520a2776a891718 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:41:46.223699 7657 raft_consensus.cc:504] T 17ee9881ee224beb8363944baf158721 P 2d027e7df09047df8520a2776a891718 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d027e7df09047df8520a2776a891718" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35637 } } peers { permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 } } peers { permanent_uuid: "cd6a34c3397b44ce85248a5a9c85f066" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33533 } }
I20260620 18:41:46.225415 7657 leader_election.cc:288] T 17ee9881ee224beb8363944baf158721 P 2d027e7df09047df8520a2776a891718 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c20fc465cf9f4ea7b5cddf62855dc2bd (127.5.225.131:46683), cd6a34c3397b44ce85248a5a9c85f066 (127.5.225.130:33533)
I20260620 18:41:46.232889 7646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:46.226005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:34395 (local address 127.5.225.130:33533)
0620 18:41:46.226251 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:46.226259 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:46.226901 (+ 642us) server_negotiation.cc:407] Connection header received
0620 18:41:46.227052 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:46.227057 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:46.227131 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:46.227220 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:46.227975 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.228952 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.231179 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.231949 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.232150 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:46.232225 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:46.232489 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 18:41:46.232613 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:41:46.232925 7649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:46.225813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:50185 (local address 127.5.225.131:46683)
0620 18:41:46.226036 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:46.226042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:46.226590 (+ 548us) server_negotiation.cc:407] Connection header received
0620 18:41:46.226822 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:46.226826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:46.226925 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:46.227026 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:46.227844 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.228811 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.231215 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.231987 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.232197 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:46.232274 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:46.232541 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 18:41:46.232670 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:41:46.233028 7660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:46.225892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.130:33533 (local address 127.5.225.129:34395)
0620 18:41:46.226786 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:46.226801 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:46.226920 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:46.227255 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:46.227261 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:46.227283 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:46.227805 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:46.227816 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.229135 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.229140 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:46.231014 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:46.231027 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.232176 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.232182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:46.232330 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:46.232353 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 18:41:46.232464 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 18:41:46.232597 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"mutex_wait_us":61,"thread_start_us":554,"threads_started":1}
I20260620 18:41:46.233471 7659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:46.225701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.131:46683 (local address 127.5.225.129:50185)
0620 18:41:46.226440 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:46.226456 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:46.226660 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:46.227069 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:46.227072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:46.227095 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:46.227643 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:46.227658 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.228935 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.228943 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:46.231049 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:46.231062 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.232227 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.232234 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:46.232385 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:46.232407 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 18:41:46.232513 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 18:41:46.232648 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"mutex_wait_us":149,"thread_start_us":341,"threads_started":1}
I20260620 18:41:46.234452 7447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ee9881ee224beb8363944baf158721" candidate_uuid: "2d027e7df09047df8520a2776a891718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd6a34c3397b44ce85248a5a9c85f066" is_pre_election: true
I20260620 18:41:46.234637 7586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ee9881ee224beb8363944baf158721" candidate_uuid: "2d027e7df09047df8520a2776a891718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" is_pre_election: true
I20260620 18:41:46.234982 7447 raft_consensus.cc:2454] T 17ee9881ee224beb8363944baf158721 P cd6a34c3397b44ce85248a5a9c85f066 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d027e7df09047df8520a2776a891718 in term 0.
I20260620 18:41:46.235186 7586 raft_consensus.cc:2454] T 17ee9881ee224beb8363944baf158721 P c20fc465cf9f4ea7b5cddf62855dc2bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d027e7df09047df8520a2776a891718 in term 0.
W20260620 18:41:46.235467 7632 tablet.cc:2369] T 17ee9881ee224beb8363944baf158721 P c20fc465cf9f4ea7b5cddf62855dc2bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:41:46.235759 7235 leader_election.cc:302] T 17ee9881ee224beb8363944baf158721 P 2d027e7df09047df8520a2776a891718 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d027e7df09047df8520a2776a891718, cd6a34c3397b44ce85248a5a9c85f066; no voters:
I20260620 18:41:46.236227 7657 raft_consensus.cc:2781] T 17ee9881ee224beb8363944baf158721 P 2d027e7df09047df8520a2776a891718 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:41:46.236393 7657 raft_consensus.cc:482] T 17ee9881ee224beb8363944baf158721 P 2d027e7df09047df8520a2776a891718 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:41:46.236503 7657 raft_consensus.cc:3037] T 17ee9881ee224beb8363944baf158721 P 2d027e7df09047df8520a2776a891718 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:46.238852 7657 raft_consensus.cc:504] T 17ee9881ee224beb8363944baf158721 P 2d027e7df09047df8520a2776a891718 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d027e7df09047df8520a2776a891718" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35637 } } peers { permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 } } peers { permanent_uuid: "cd6a34c3397b44ce85248a5a9c85f066" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33533 } }
I20260620 18:41:46.239473 7657 leader_election.cc:288] T 17ee9881ee224beb8363944baf158721 P 2d027e7df09047df8520a2776a891718 [CANDIDATE]: Term 1 election: Requested vote from peers c20fc465cf9f4ea7b5cddf62855dc2bd (127.5.225.131:46683), cd6a34c3397b44ce85248a5a9c85f066 (127.5.225.130:33533)
I20260620 18:41:46.239869 7586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ee9881ee224beb8363944baf158721" candidate_uuid: "2d027e7df09047df8520a2776a891718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd"
I20260620 18:41:46.239928 7447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ee9881ee224beb8363944baf158721" candidate_uuid: "2d027e7df09047df8520a2776a891718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd6a34c3397b44ce85248a5a9c85f066"
I20260620 18:41:46.240089 7586 raft_consensus.cc:3037] T 17ee9881ee224beb8363944baf158721 P c20fc465cf9f4ea7b5cddf62855dc2bd [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:46.240110 7447 raft_consensus.cc:3037] T 17ee9881ee224beb8363944baf158721 P cd6a34c3397b44ce85248a5a9c85f066 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:46.243289 7586 raft_consensus.cc:2454] T 17ee9881ee224beb8363944baf158721 P c20fc465cf9f4ea7b5cddf62855dc2bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d027e7df09047df8520a2776a891718 in term 1.
I20260620 18:41:46.243289 7447 raft_consensus.cc:2454] T 17ee9881ee224beb8363944baf158721 P cd6a34c3397b44ce85248a5a9c85f066 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d027e7df09047df8520a2776a891718 in term 1.
I20260620 18:41:46.243728 7233 leader_election.cc:302] T 17ee9881ee224beb8363944baf158721 P 2d027e7df09047df8520a2776a891718 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d027e7df09047df8520a2776a891718, c20fc465cf9f4ea7b5cddf62855dc2bd; no voters:
I20260620 18:41:46.244074 7657 raft_consensus.cc:2781] T 17ee9881ee224beb8363944baf158721 P 2d027e7df09047df8520a2776a891718 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:41:46.244613 7657 raft_consensus.cc:686] T 17ee9881ee224beb8363944baf158721 P 2d027e7df09047df8520a2776a891718 [term 1 LEADER]: Becoming Leader. State: Replica: 2d027e7df09047df8520a2776a891718, State: Running, Role: LEADER
I20260620 18:41:46.245087 7657 consensus_queue.cc:237] T 17ee9881ee224beb8363944baf158721 P 2d027e7df09047df8520a2776a891718 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d027e7df09047df8520a2776a891718" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35637 } } peers { permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 } } peers { permanent_uuid: "cd6a34c3397b44ce85248a5a9c85f066" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33533 } }
I20260620 18:41:46.249765 7145 catalog_manager.cc:5495] T 17ee9881ee224beb8363944baf158721 P 2d027e7df09047df8520a2776a891718 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d027e7df09047df8520a2776a891718 (127.5.225.129). New cstate: current_term: 1 leader_uuid: "2d027e7df09047df8520a2776a891718" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d027e7df09047df8520a2776a891718" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35637 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd6a34c3397b44ce85248a5a9c85f066" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33533 } health_report { overall_health: UNKNOWN } } }
I20260620 18:41:46.283921 7645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:46.278642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58976 (local address 127.5.225.129:35637)
0620 18:41:46.278891 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:46.278899 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:46.279019 (+ 120us) server_negotiation.cc:407] Connection header received
0620 18:41:46.279268 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:46.279272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:46.279348 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:46.279467 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:46.279767 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.280453 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.281751 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.282216 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.282321 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:46.282801 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:46.283303 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:46.283407 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:46.283558 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 18:41:46.283618 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 18:41:46.292060 7586 raft_consensus.cc:1261] T 17ee9881ee224beb8363944baf158721 P c20fc465cf9f4ea7b5cddf62855dc2bd [term 1 FOLLOWER]: Refusing update from remote peer 2d027e7df09047df8520a2776a891718: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:41:46.292060 7447 raft_consensus.cc:1261] T 17ee9881ee224beb8363944baf158721 P cd6a34c3397b44ce85248a5a9c85f066 [term 1 FOLLOWER]: Refusing update from remote peer 2d027e7df09047df8520a2776a891718: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:41:46.293047 7661 consensus_queue.cc:1035] T 17ee9881ee224beb8363944baf158721 P 2d027e7df09047df8520a2776a891718 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd6a34c3397b44ce85248a5a9c85f066" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:41:46.293426 7657 consensus_queue.cc:1035] T 17ee9881ee224beb8363944baf158721 P 2d027e7df09047df8520a2776a891718 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:41:46.324262 7146 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:41:46.324553 7146 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:41:46.328091 7146 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f63f8e6e7e064537a0e37780fdbab743: Sending DeleteTablet for 3 replicas of tablet 17ee9881ee224beb8363944baf158721
I20260620 18:41:46.329349 7425 tablet_service.cc:1514] Processing DeleteTablet for tablet 17ee9881ee224beb8363944baf158721 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:41:46 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:51856
I20260620 18:41:46.329350 7283 tablet_service.cc:1514] Processing DeleteTablet for tablet 17ee9881ee224beb8363944baf158721 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:41:46 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:58966
I20260620 18:41:46.329367 7563 tablet_service.cc:1514] Processing DeleteTablet for tablet 17ee9881ee224beb8363944baf158721 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:41:46 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:59066
I20260620 18:41:46.331573 7146 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:41:46.333395 7146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:41:46.343477 7674 tablet_replica.cc:330] T 17ee9881ee224beb8363944baf158721 P c20fc465cf9f4ea7b5cddf62855dc2bd: stopping tablet replica
I20260620 18:41:46.343955 7674 raft_consensus.cc:2229] T 17ee9881ee224beb8363944baf158721 P c20fc465cf9f4ea7b5cddf62855dc2bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:41:46.344480 7674 raft_consensus.cc:2258] T 17ee9881ee224beb8363944baf158721 P c20fc465cf9f4ea7b5cddf62855dc2bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:41:46.345084 7423 tablet_service.cc:1467] Processing CreateTablet for tablet fddde2e47e5246c2933a1ffda90d6713 (DEFAULT_TABLE table=test-table [id=cd492c4e6c6a40bbaa55dabf29c9560e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:41:46.345839 7423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fddde2e47e5246c2933a1ffda90d6713. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:46.348188 7282 tablet_service.cc:1467] Processing CreateTablet for tablet fddde2e47e5246c2933a1ffda90d6713 (DEFAULT_TABLE table=test-table [id=cd492c4e6c6a40bbaa55dabf29c9560e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:41:46.348829 7282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fddde2e47e5246c2933a1ffda90d6713. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:46.350910 7675 tablet_replica.cc:330] T 17ee9881ee224beb8363944baf158721 P cd6a34c3397b44ce85248a5a9c85f066: stopping tablet replica
I20260620 18:41:46.351102 7560 tablet_service.cc:1467] Processing CreateTablet for tablet fddde2e47e5246c2933a1ffda90d6713 (DEFAULT_TABLE table=test-table [id=cd492c4e6c6a40bbaa55dabf29c9560e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:41:46.351317 7675 raft_consensus.cc:2229] T 17ee9881ee224beb8363944baf158721 P cd6a34c3397b44ce85248a5a9c85f066 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:41:46.351514 7560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fddde2e47e5246c2933a1ffda90d6713. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:46.351758 7675 raft_consensus.cc:2258] T 17ee9881ee224beb8363944baf158721 P cd6a34c3397b44ce85248a5a9c85f066 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:41:46.353443 7674 ts_tablet_manager.cc:1905] T 17ee9881ee224beb8363944baf158721 P c20fc465cf9f4ea7b5cddf62855dc2bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:41:46.360184 7674 ts_tablet_manager.cc:1918] T 17ee9881ee224beb8363944baf158721 P c20fc465cf9f4ea7b5cddf62855dc2bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:41:46.360324 7674 log.cc:1196] T 17ee9881ee224beb8363944baf158721 P c20fc465cf9f4ea7b5cddf62855dc2bd: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-2/wal/wals/17ee9881ee224beb8363944baf158721
I20260620 18:41:46.360816 7674 ts_tablet_manager.cc:1939] T 17ee9881ee224beb8363944baf158721 P c20fc465cf9f4ea7b5cddf62855dc2bd: Deleting consensus metadata
I20260620 18:41:46.350478 7676 tablet_replica.cc:330] T 17ee9881ee224beb8363944baf158721 P 2d027e7df09047df8520a2776a891718: stopping tablet replica
I20260620 18:41:46.361393 7676 raft_consensus.cc:2229] T 17ee9881ee224beb8363944baf158721 P 2d027e7df09047df8520a2776a891718 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:41:46.362015 7676 raft_consensus.cc:2258] T 17ee9881ee224beb8363944baf158721 P 2d027e7df09047df8520a2776a891718 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:41:46.363003 7130 catalog_manager.cc:4841] TS c20fc465cf9f4ea7b5cddf62855dc2bd (127.5.225.131:46683): tablet 17ee9881ee224beb8363944baf158721 (table test-table [id=ba46a03d065b491eb2fbf8a1a5510a14]) successfully deleted
I20260620 18:41:46.363924 7676 ts_tablet_manager.cc:1905] T 17ee9881ee224beb8363944baf158721 P 2d027e7df09047df8520a2776a891718: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:41:46.364370 7675 ts_tablet_manager.cc:1905] T 17ee9881ee224beb8363944baf158721 P cd6a34c3397b44ce85248a5a9c85f066: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:41:46.370069 7651 tablet_bootstrap.cc:492] T fddde2e47e5246c2933a1ffda90d6713 P c20fc465cf9f4ea7b5cddf62855dc2bd: Bootstrap starting.
I20260620 18:41:46.370711 7676 ts_tablet_manager.cc:1918] T 17ee9881ee224beb8363944baf158721 P 2d027e7df09047df8520a2776a891718: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:41:46.370895 7676 log.cc:1196] T 17ee9881ee224beb8363944baf158721 P 2d027e7df09047df8520a2776a891718: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-0/wal/wals/17ee9881ee224beb8363944baf158721
I20260620 18:41:46.371479 7676 ts_tablet_manager.cc:1939] T 17ee9881ee224beb8363944baf158721 P 2d027e7df09047df8520a2776a891718: Deleting consensus metadata
I20260620 18:41:46.371728 7651 tablet_bootstrap.cc:654] T fddde2e47e5246c2933a1ffda90d6713 P c20fc465cf9f4ea7b5cddf62855dc2bd: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:46.373190 7130 catalog_manager.cc:4841] TS 2d027e7df09047df8520a2776a891718 (127.5.225.129:35637): tablet 17ee9881ee224beb8363944baf158721 (table test-table [id=ba46a03d065b491eb2fbf8a1a5510a14]) successfully deleted
I20260620 18:41:46.373868 7650 tablet_bootstrap.cc:492] T fddde2e47e5246c2933a1ffda90d6713 P 2d027e7df09047df8520a2776a891718: Bootstrap starting.
I20260620 18:41:46.374948 7652 tablet_bootstrap.cc:492] T fddde2e47e5246c2933a1ffda90d6713 P cd6a34c3397b44ce85248a5a9c85f066: Bootstrap starting.
I20260620 18:41:46.375471 7650 tablet_bootstrap.cc:654] T fddde2e47e5246c2933a1ffda90d6713 P 2d027e7df09047df8520a2776a891718: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:46.376778 7652 tablet_bootstrap.cc:654] T fddde2e47e5246c2933a1ffda90d6713 P cd6a34c3397b44ce85248a5a9c85f066: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:46.379546 7675 ts_tablet_manager.cc:1918] T 17ee9881ee224beb8363944baf158721 P cd6a34c3397b44ce85248a5a9c85f066: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:41:46.379772 7675 log.cc:1196] T 17ee9881ee224beb8363944baf158721 P cd6a34c3397b44ce85248a5a9c85f066: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-1/wal/wals/17ee9881ee224beb8363944baf158721
I20260620 18:41:46.380431 7675 ts_tablet_manager.cc:1939] T 17ee9881ee224beb8363944baf158721 P cd6a34c3397b44ce85248a5a9c85f066: Deleting consensus metadata
I20260620 18:41:46.382629 7132 catalog_manager.cc:4841] TS cd6a34c3397b44ce85248a5a9c85f066 (127.5.225.130:33533): tablet 17ee9881ee224beb8363944baf158721 (table test-table [id=ba46a03d065b491eb2fbf8a1a5510a14]) successfully deleted
I20260620 18:41:46.384174 7652 tablet_bootstrap.cc:492] T fddde2e47e5246c2933a1ffda90d6713 P cd6a34c3397b44ce85248a5a9c85f066: No bootstrap required, opened a new log
I20260620 18:41:46.384291 7651 tablet_bootstrap.cc:492] T fddde2e47e5246c2933a1ffda90d6713 P c20fc465cf9f4ea7b5cddf62855dc2bd: No bootstrap required, opened a new log
I20260620 18:41:46.384320 7652 ts_tablet_manager.cc:1397] T fddde2e47e5246c2933a1ffda90d6713 P cd6a34c3397b44ce85248a5a9c85f066: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20260620 18:41:46.384444 7651 ts_tablet_manager.cc:1397] T fddde2e47e5246c2933a1ffda90d6713 P c20fc465cf9f4ea7b5cddf62855dc2bd: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.007s
I20260620 18:41:46.385102 7650 tablet_bootstrap.cc:492] T fddde2e47e5246c2933a1ffda90d6713 P 2d027e7df09047df8520a2776a891718: No bootstrap required, opened a new log
I20260620 18:41:46.385118 7652 raft_consensus.cc:348] T fddde2e47e5246c2933a1ffda90d6713 P cd6a34c3397b44ce85248a5a9c85f066 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d027e7df09047df8520a2776a891718" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35637 } } peers { permanent_uuid: "cd6a34c3397b44ce85248a5a9c85f066" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33533 } } peers { permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 } }
I20260620 18:41:46.385300 7650 ts_tablet_manager.cc:1397] T fddde2e47e5246c2933a1ffda90d6713 P 2d027e7df09047df8520a2776a891718: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260620 18:41:46.385392 7652 raft_consensus.cc:374] T fddde2e47e5246c2933a1ffda90d6713 P cd6a34c3397b44ce85248a5a9c85f066 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:46.385252 7651 raft_consensus.cc:348] T fddde2e47e5246c2933a1ffda90d6713 P c20fc465cf9f4ea7b5cddf62855dc2bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d027e7df09047df8520a2776a891718" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35637 } } peers { permanent_uuid: "cd6a34c3397b44ce85248a5a9c85f066" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33533 } } peers { permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 } }
I20260620 18:41:46.385516 7651 raft_consensus.cc:374] T fddde2e47e5246c2933a1ffda90d6713 P c20fc465cf9f4ea7b5cddf62855dc2bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:46.385525 7652 raft_consensus.cc:729] T fddde2e47e5246c2933a1ffda90d6713 P cd6a34c3397b44ce85248a5a9c85f066 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd6a34c3397b44ce85248a5a9c85f066, State: Initialized, Role: FOLLOWER
I20260620 18:41:46.385618 7651 raft_consensus.cc:729] T fddde2e47e5246c2933a1ffda90d6713 P c20fc465cf9f4ea7b5cddf62855dc2bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c20fc465cf9f4ea7b5cddf62855dc2bd, State: Initialized, Role: FOLLOWER
I20260620 18:41:46.385793 7652 consensus_queue.cc:260] T fddde2e47e5246c2933a1ffda90d6713 P cd6a34c3397b44ce85248a5a9c85f066 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d027e7df09047df8520a2776a891718" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35637 } } peers { permanent_uuid: "cd6a34c3397b44ce85248a5a9c85f066" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33533 } } peers { permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 } }
I20260620 18:41:46.385897 7651 consensus_queue.cc:260] T fddde2e47e5246c2933a1ffda90d6713 P c20fc465cf9f4ea7b5cddf62855dc2bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d027e7df09047df8520a2776a891718" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35637 } } peers { permanent_uuid: "cd6a34c3397b44ce85248a5a9c85f066" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33533 } } peers { permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 } }
I20260620 18:41:46.386142 7650 raft_consensus.cc:348] T fddde2e47e5246c2933a1ffda90d6713 P 2d027e7df09047df8520a2776a891718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d027e7df09047df8520a2776a891718" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35637 } } peers { permanent_uuid: "cd6a34c3397b44ce85248a5a9c85f066" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33533 } } peers { permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 } }
I20260620 18:41:46.386406 7652 ts_tablet_manager.cc:1428] T fddde2e47e5246c2933a1ffda90d6713 P cd6a34c3397b44ce85248a5a9c85f066: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:41:46.386392 7650 raft_consensus.cc:374] T fddde2e47e5246c2933a1ffda90d6713 P 2d027e7df09047df8520a2776a891718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:46.386547 7650 raft_consensus.cc:729] T fddde2e47e5246c2933a1ffda90d6713 P 2d027e7df09047df8520a2776a891718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d027e7df09047df8520a2776a891718, State: Initialized, Role: FOLLOWER
I20260620 18:41:46.386819 7650 consensus_queue.cc:260] T fddde2e47e5246c2933a1ffda90d6713 P 2d027e7df09047df8520a2776a891718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d027e7df09047df8520a2776a891718" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35637 } } peers { permanent_uuid: "cd6a34c3397b44ce85248a5a9c85f066" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33533 } } peers { permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 } }
I20260620 18:41:46.387370 7650 ts_tablet_manager.cc:1428] T fddde2e47e5246c2933a1ffda90d6713 P 2d027e7df09047df8520a2776a891718: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260620 18:41:46.388039 7651 ts_tablet_manager.cc:1428] T fddde2e47e5246c2933a1ffda90d6713 P c20fc465cf9f4ea7b5cddf62855dc2bd: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260620 18:41:46.565565 7658 raft_consensus.cc:482] T fddde2e47e5246c2933a1ffda90d6713 P cd6a34c3397b44ce85248a5a9c85f066 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:41:46.565840 7658 raft_consensus.cc:504] T fddde2e47e5246c2933a1ffda90d6713 P cd6a34c3397b44ce85248a5a9c85f066 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d027e7df09047df8520a2776a891718" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35637 } } peers { permanent_uuid: "cd6a34c3397b44ce85248a5a9c85f066" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33533 } } peers { permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 } }
I20260620 18:41:46.567498 7658 leader_election.cc:288] T fddde2e47e5246c2933a1ffda90d6713 P cd6a34c3397b44ce85248a5a9c85f066 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d027e7df09047df8520a2776a891718 (127.5.225.129:35637), c20fc465cf9f4ea7b5cddf62855dc2bd (127.5.225.131:46683)
I20260620 18:41:46.574818 7645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:46.567874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:47779 (local address 127.5.225.129:35637)
0620 18:41:46.568081 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:46.568088 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:46.568438 (+ 350us) server_negotiation.cc:407] Connection header received
0620 18:41:46.568685 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:46.568694 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:46.568859 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:46.568943 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:46.569812 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.571052 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.572881 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.573738 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.573900 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:46.573975 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:46.574350 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 18:41:46.574485 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 18:41:46.574844 7680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:46.567763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:35637 (local address 127.5.225.130:47779)
0620 18:41:46.568317 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:46.568339 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:46.568482 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:46.569201 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:46.569211 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:46.569243 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:46.569593 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:46.569609 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.571196 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.571202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:46.572725 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:46.572737 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.573952 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.573958 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:46.574138 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:46.574208 (+ 70us) client_negotiation.cc:769] Sending connection context
0620 18:41:46.574310 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 18:41:46.574446 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":24,"thread_start_us":280,"threads_started":1}
I20260620 18:41:46.576050 7681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:46.568486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.131:46683 (local address 127.5.225.130:46361)
0620 18:41:46.569409 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:46.569433 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:46.569588 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:46.569950 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:46.569953 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:46.569977 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:46.570358 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:46.570367 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.571405 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.571410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:46.573543 (+ 2133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:46.573557 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.575275 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.575282 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:46.575462 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:46.575493 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 18:41:46.575586 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 18:41:46.575722 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"thread_start_us":559,"threads_started":1}
I20260620 18:41:46.575804 7303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fddde2e47e5246c2933a1ffda90d6713" candidate_uuid: "cd6a34c3397b44ce85248a5a9c85f066" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d027e7df09047df8520a2776a891718" is_pre_election: true
I20260620 18:41:46.576267 7649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:46.568715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:46361 (local address 127.5.225.131:46683)
0620 18:41:46.568896 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:46.568903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:46.569560 (+ 657us) server_negotiation.cc:407] Connection header received
0620 18:41:46.569745 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:46.569752 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:46.569832 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:46.569930 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:46.570510 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.571249 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.573706 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.574850 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.575017 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:46.575103 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:46.575864 (+ 761us) server_negotiation.cc:299] Negotiation successful
0620 18:41:46.576025 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:41:46.576316 7303 raft_consensus.cc:2454] T fddde2e47e5246c2933a1ffda90d6713 P 2d027e7df09047df8520a2776a891718 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd6a34c3397b44ce85248a5a9c85f066 in term 0.
I20260620 18:41:46.576874 7586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fddde2e47e5246c2933a1ffda90d6713" candidate_uuid: "cd6a34c3397b44ce85248a5a9c85f066" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" is_pre_election: true
I20260620 18:41:46.576994 7373 leader_election.cc:302] T fddde2e47e5246c2933a1ffda90d6713 P cd6a34c3397b44ce85248a5a9c85f066 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d027e7df09047df8520a2776a891718, cd6a34c3397b44ce85248a5a9c85f066; no voters:
I20260620 18:41:46.577139 7586 raft_consensus.cc:2454] T fddde2e47e5246c2933a1ffda90d6713 P c20fc465cf9f4ea7b5cddf62855dc2bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd6a34c3397b44ce85248a5a9c85f066 in term 0.
I20260620 18:41:46.577407 7658 raft_consensus.cc:2781] T fddde2e47e5246c2933a1ffda90d6713 P cd6a34c3397b44ce85248a5a9c85f066 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:41:46.577531 7658 raft_consensus.cc:482] T fddde2e47e5246c2933a1ffda90d6713 P cd6a34c3397b44ce85248a5a9c85f066 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:41:46.577630 7658 raft_consensus.cc:3037] T fddde2e47e5246c2933a1ffda90d6713 P cd6a34c3397b44ce85248a5a9c85f066 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:46.579950 7658 raft_consensus.cc:504] T fddde2e47e5246c2933a1ffda90d6713 P cd6a34c3397b44ce85248a5a9c85f066 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d027e7df09047df8520a2776a891718" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35637 } } peers { permanent_uuid: "cd6a34c3397b44ce85248a5a9c85f066" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33533 } } peers { permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 } }
I20260620 18:41:46.580569 7658 leader_election.cc:288] T fddde2e47e5246c2933a1ffda90d6713 P cd6a34c3397b44ce85248a5a9c85f066 [CANDIDATE]: Term 1 election: Requested vote from peers 2d027e7df09047df8520a2776a891718 (127.5.225.129:35637), c20fc465cf9f4ea7b5cddf62855dc2bd (127.5.225.131:46683)
I20260620 18:41:46.581028 7303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fddde2e47e5246c2933a1ffda90d6713" candidate_uuid: "cd6a34c3397b44ce85248a5a9c85f066" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d027e7df09047df8520a2776a891718"
I20260620 18:41:46.581123 7586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fddde2e47e5246c2933a1ffda90d6713" candidate_uuid: "cd6a34c3397b44ce85248a5a9c85f066" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd"
I20260620 18:41:46.581244 7303 raft_consensus.cc:3037] T fddde2e47e5246c2933a1ffda90d6713 P 2d027e7df09047df8520a2776a891718 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:46.581317 7586 raft_consensus.cc:3037] T fddde2e47e5246c2933a1ffda90d6713 P c20fc465cf9f4ea7b5cddf62855dc2bd [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:46.584329 7586 raft_consensus.cc:2454] T fddde2e47e5246c2933a1ffda90d6713 P c20fc465cf9f4ea7b5cddf62855dc2bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd6a34c3397b44ce85248a5a9c85f066 in term 1.
I20260620 18:41:46.584553 7303 raft_consensus.cc:2454] T fddde2e47e5246c2933a1ffda90d6713 P 2d027e7df09047df8520a2776a891718 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd6a34c3397b44ce85248a5a9c85f066 in term 1.
I20260620 18:41:46.584762 7373 leader_election.cc:302] T fddde2e47e5246c2933a1ffda90d6713 P cd6a34c3397b44ce85248a5a9c85f066 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c20fc465cf9f4ea7b5cddf62855dc2bd, cd6a34c3397b44ce85248a5a9c85f066; no voters:
I20260620 18:41:46.585063 7658 raft_consensus.cc:2781] T fddde2e47e5246c2933a1ffda90d6713 P cd6a34c3397b44ce85248a5a9c85f066 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:41:46.585542 7658 raft_consensus.cc:686] T fddde2e47e5246c2933a1ffda90d6713 P cd6a34c3397b44ce85248a5a9c85f066 [term 1 LEADER]: Becoming Leader. State: Replica: cd6a34c3397b44ce85248a5a9c85f066, State: Running, Role: LEADER
I20260620 18:41:46.585958 7658 consensus_queue.cc:237] T fddde2e47e5246c2933a1ffda90d6713 P cd6a34c3397b44ce85248a5a9c85f066 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d027e7df09047df8520a2776a891718" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35637 } } peers { permanent_uuid: "cd6a34c3397b44ce85248a5a9c85f066" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33533 } } peers { permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 } }
I20260620 18:41:46.589448 7144 catalog_manager.cc:5495] T fddde2e47e5246c2933a1ffda90d6713 P cd6a34c3397b44ce85248a5a9c85f066 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd6a34c3397b44ce85248a5a9c85f066 (127.5.225.130). New cstate: current_term: 1 leader_uuid: "cd6a34c3397b44ce85248a5a9c85f066" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d027e7df09047df8520a2776a891718" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd6a34c3397b44ce85248a5a9c85f066" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 } health_report { overall_health: UNKNOWN } } }
Jun 20 18:41:46 dist-test-slave-k6lg krb5kdc[7101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980904, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:41:46.644274 7690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:46.628673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:44115 (local address 127.0.0.1:38436)
0620 18:41:46.629714 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:46.629729 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:46.629895 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:46.630282 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:46.630287 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:46.631050 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:46.631274 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:46.631284 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.632375 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.632380 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:46.633449 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:46.633475 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.634512 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.634521 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:46.634671 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:46.635358 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:46.635379 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:46.637877 (+ 2498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:46.641713 (+ 3836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:46.641722 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:46.641733 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:46.642130 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:46.642583 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:46.642586 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:46.642588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:46.642711 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:46.643218 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:46.643226 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:46.643383 (+ 157us) client_negotiation.cc:769] Sending connection context
0620 18:41:46.643551 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 18:41:46.643822 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":902,"thread_start_us":757,"threads_started":1}
I20260620 18:41:46.644332 7689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:46.628672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38436 (local address 127.5.225.190:44115)
0620 18:41:46.629563 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:46.629571 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:46.629816 (+ 245us) server_negotiation.cc:407] Connection header received
0620 18:41:46.630015 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:46.630025 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:46.630119 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:46.630253 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:46.631447 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.632242 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.633625 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.634319 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.634473 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:46.638101 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:46.638109 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:46.638114 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:46.638150 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:46.641575 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:46.642310 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:46.642317 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:46.642321 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:46.642446 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:46.642809 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:46.642816 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:46.642820 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:46.643087 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:46.643230 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:46.643633 (+ 403us) server_negotiation.cc:299] Negotiation successful
0620 18:41:46.643948 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":127,"server-negotiator.queue_time_us":651,"thread_start_us":518,"threads_started":1}
I20260620 18:41:46.649852 7144 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:38436:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 18:41:46.651147 7144 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:41:46.658111 7282 tablet_service.cc:1467] Processing CreateTablet for tablet 743cdbaf7d7d4eeb88ed6ffba18ac9d9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=26ec1c033ab746a1875bdb7c1a9b6d16]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:41:46.658269 7423 tablet_service.cc:1467] Processing CreateTablet for tablet 743cdbaf7d7d4eeb88ed6ffba18ac9d9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=26ec1c033ab746a1875bdb7c1a9b6d16]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:41:46.658756 7423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 743cdbaf7d7d4eeb88ed6ffba18ac9d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:46.658764 7282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 743cdbaf7d7d4eeb88ed6ffba18ac9d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:46.658602 7563 tablet_service.cc:1467] Processing CreateTablet for tablet 743cdbaf7d7d4eeb88ed6ffba18ac9d9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=26ec1c033ab746a1875bdb7c1a9b6d16]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:41:46.659180 7563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 743cdbaf7d7d4eeb88ed6ffba18ac9d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:46.665953 7652 tablet_bootstrap.cc:492] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P cd6a34c3397b44ce85248a5a9c85f066: Bootstrap starting.
I20260620 18:41:46.666293 7650 tablet_bootstrap.cc:492] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P 2d027e7df09047df8520a2776a891718: Bootstrap starting.
I20260620 18:41:46.667728 7652 tablet_bootstrap.cc:654] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P cd6a34c3397b44ce85248a5a9c85f066: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:46.668016 7650 tablet_bootstrap.cc:654] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P 2d027e7df09047df8520a2776a891718: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:46.668838 7651 tablet_bootstrap.cc:492] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P c20fc465cf9f4ea7b5cddf62855dc2bd: Bootstrap starting.
I20260620 18:41:46.671015 7651 tablet_bootstrap.cc:654] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P c20fc465cf9f4ea7b5cddf62855dc2bd: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:46.672276 7650 tablet_bootstrap.cc:492] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P 2d027e7df09047df8520a2776a891718: No bootstrap required, opened a new log
I20260620 18:41:46.672489 7650 ts_tablet_manager.cc:1397] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P 2d027e7df09047df8520a2776a891718: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260620 18:41:46.673429 7652 tablet_bootstrap.cc:492] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P cd6a34c3397b44ce85248a5a9c85f066: No bootstrap required, opened a new log
I20260620 18:41:46.673355 7650 raft_consensus.cc:348] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P 2d027e7df09047df8520a2776a891718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d027e7df09047df8520a2776a891718" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35637 } } peers { permanent_uuid: "cd6a34c3397b44ce85248a5a9c85f066" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33533 } } peers { permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 } }
I20260620 18:41:46.673601 7652 ts_tablet_manager.cc:1397] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P cd6a34c3397b44ce85248a5a9c85f066: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260620 18:41:46.673606 7650 raft_consensus.cc:374] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P 2d027e7df09047df8520a2776a891718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:46.673691 7650 raft_consensus.cc:729] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P 2d027e7df09047df8520a2776a891718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d027e7df09047df8520a2776a891718, State: Initialized, Role: FOLLOWER
I20260620 18:41:46.673944 7650 consensus_queue.cc:260] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P 2d027e7df09047df8520a2776a891718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d027e7df09047df8520a2776a891718" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35637 } } peers { permanent_uuid: "cd6a34c3397b44ce85248a5a9c85f066" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33533 } } peers { permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 } }
I20260620 18:41:46.674360 7652 raft_consensus.cc:348] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P cd6a34c3397b44ce85248a5a9c85f066 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d027e7df09047df8520a2776a891718" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35637 } } peers { permanent_uuid: "cd6a34c3397b44ce85248a5a9c85f066" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33533 } } peers { permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 } }
I20260620 18:41:46.674432 7657 tablet_replica.cc:439] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P 2d027e7df09047df8520a2776a891718: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d027e7df09047df8520a2776a891718" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35637 } } peers { permanent_uuid: "cd6a34c3397b44ce85248a5a9c85f066" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33533 } } peers { permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 } } }
I20260620 18:41:46.674608 7652 raft_consensus.cc:374] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P cd6a34c3397b44ce85248a5a9c85f066 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:46.674712 7657 tablet_replica.cc:442] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P 2d027e7df09047df8520a2776a891718: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:41:46.674676 7652 raft_consensus.cc:729] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P cd6a34c3397b44ce85248a5a9c85f066 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd6a34c3397b44ce85248a5a9c85f066, State: Initialized, Role: FOLLOWER
I20260620 18:41:46.675172 7652 consensus_queue.cc:260] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P cd6a34c3397b44ce85248a5a9c85f066 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d027e7df09047df8520a2776a891718" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35637 } } peers { permanent_uuid: "cd6a34c3397b44ce85248a5a9c85f066" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33533 } } peers { permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 } }
I20260620 18:41:46.676369 7650 ts_tablet_manager.cc:1428] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P 2d027e7df09047df8520a2776a891718: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 18:41:46.677863 7658 tablet_replica.cc:439] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P cd6a34c3397b44ce85248a5a9c85f066: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d027e7df09047df8520a2776a891718" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35637 } } peers { permanent_uuid: "cd6a34c3397b44ce85248a5a9c85f066" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33533 } } peers { permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 } } }
I20260620 18:41:46.678210 7658 tablet_replica.cc:442] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P cd6a34c3397b44ce85248a5a9c85f066: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:41:46.678907 7652 ts_tablet_manager.cc:1428] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P cd6a34c3397b44ce85248a5a9c85f066: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260620 18:41:46.679266 7651 tablet_bootstrap.cc:492] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P c20fc465cf9f4ea7b5cddf62855dc2bd: No bootstrap required, opened a new log
I20260620 18:41:46.679761 7651 ts_tablet_manager.cc:1397] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P c20fc465cf9f4ea7b5cddf62855dc2bd: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260620 18:41:46.680606 7651 raft_consensus.cc:348] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P c20fc465cf9f4ea7b5cddf62855dc2bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d027e7df09047df8520a2776a891718" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35637 } } peers { permanent_uuid: "cd6a34c3397b44ce85248a5a9c85f066" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33533 } } peers { permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 } }
I20260620 18:41:46.680923 7651 raft_consensus.cc:374] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P c20fc465cf9f4ea7b5cddf62855dc2bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:46.681046 7651 raft_consensus.cc:729] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P c20fc465cf9f4ea7b5cddf62855dc2bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c20fc465cf9f4ea7b5cddf62855dc2bd, State: Initialized, Role: FOLLOWER
I20260620 18:41:46.681320 7651 consensus_queue.cc:260] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P c20fc465cf9f4ea7b5cddf62855dc2bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d027e7df09047df8520a2776a891718" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35637 } } peers { permanent_uuid: "cd6a34c3397b44ce85248a5a9c85f066" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33533 } } peers { permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 } }
I20260620 18:41:46.681871 7651 ts_tablet_manager.cc:1428] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P c20fc465cf9f4ea7b5cddf62855dc2bd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:41:46.681800 7656 tablet_replica.cc:439] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P c20fc465cf9f4ea7b5cddf62855dc2bd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d027e7df09047df8520a2776a891718" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35637 } } peers { permanent_uuid: "cd6a34c3397b44ce85248a5a9c85f066" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33533 } } peers { permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 } } }
I20260620 18:41:46.682188 7656 tablet_replica.cc:442] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P c20fc465cf9f4ea7b5cddf62855dc2bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:41:46.682713 7657 raft_consensus.cc:482] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P 2d027e7df09047df8520a2776a891718 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:41:46.682868 7657 raft_consensus.cc:504] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P 2d027e7df09047df8520a2776a891718 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d027e7df09047df8520a2776a891718" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35637 } } peers { permanent_uuid: "cd6a34c3397b44ce85248a5a9c85f066" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33533 } } peers { permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 } }
I20260620 18:41:46.683555 7657 leader_election.cc:288] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P 2d027e7df09047df8520a2776a891718 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd6a34c3397b44ce85248a5a9c85f066 (127.5.225.130:33533), c20fc465cf9f4ea7b5cddf62855dc2bd (127.5.225.131:46683)
I20260620 18:41:46.683959 7447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "743cdbaf7d7d4eeb88ed6ffba18ac9d9" candidate_uuid: "2d027e7df09047df8520a2776a891718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd6a34c3397b44ce85248a5a9c85f066" is_pre_election: true
I20260620 18:41:46.684190 7447 raft_consensus.cc:2454] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P cd6a34c3397b44ce85248a5a9c85f066 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d027e7df09047df8520a2776a891718 in term 0.
I20260620 18:41:46.684499 7586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "743cdbaf7d7d4eeb88ed6ffba18ac9d9" candidate_uuid: "2d027e7df09047df8520a2776a891718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" is_pre_election: true
I20260620 18:41:46.684732 7586 raft_consensus.cc:2454] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P c20fc465cf9f4ea7b5cddf62855dc2bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d027e7df09047df8520a2776a891718 in term 0.
I20260620 18:41:46.685146 7233 leader_election.cc:302] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P 2d027e7df09047df8520a2776a891718 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2d027e7df09047df8520a2776a891718, c20fc465cf9f4ea7b5cddf62855dc2bd, cd6a34c3397b44ce85248a5a9c85f066; no voters:
I20260620 18:41:46.685436 7657 raft_consensus.cc:2781] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P 2d027e7df09047df8520a2776a891718 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:41:46.685555 7657 raft_consensus.cc:482] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P 2d027e7df09047df8520a2776a891718 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:41:46.685662 7657 raft_consensus.cc:3037] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P 2d027e7df09047df8520a2776a891718 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:46.688122 7657 raft_consensus.cc:504] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P 2d027e7df09047df8520a2776a891718 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d027e7df09047df8520a2776a891718" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35637 } } peers { permanent_uuid: "cd6a34c3397b44ce85248a5a9c85f066" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33533 } } peers { permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 } }
I20260620 18:41:46.689065 7657 leader_election.cc:288] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P 2d027e7df09047df8520a2776a891718 [CANDIDATE]: Term 1 election: Requested vote from peers cd6a34c3397b44ce85248a5a9c85f066 (127.5.225.130:33533), c20fc465cf9f4ea7b5cddf62855dc2bd (127.5.225.131:46683)
I20260620 18:41:46.689167 7586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "743cdbaf7d7d4eeb88ed6ffba18ac9d9" candidate_uuid: "2d027e7df09047df8520a2776a891718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd"
I20260620 18:41:46.689393 7586 raft_consensus.cc:3037] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P c20fc465cf9f4ea7b5cddf62855dc2bd [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:46.692293 7586 raft_consensus.cc:2454] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P c20fc465cf9f4ea7b5cddf62855dc2bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d027e7df09047df8520a2776a891718 in term 1.
I20260620 18:41:46.692694 7447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "743cdbaf7d7d4eeb88ed6ffba18ac9d9" candidate_uuid: "2d027e7df09047df8520a2776a891718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd6a34c3397b44ce85248a5a9c85f066"
I20260620 18:41:46.692776 7233 leader_election.cc:302] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P 2d027e7df09047df8520a2776a891718 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d027e7df09047df8520a2776a891718, c20fc465cf9f4ea7b5cddf62855dc2bd; no voters:
I20260620 18:41:46.692911 7447 raft_consensus.cc:3037] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P cd6a34c3397b44ce85248a5a9c85f066 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:46.693092 7657 raft_consensus.cc:2781] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P 2d027e7df09047df8520a2776a891718 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:41:46.693264 7657 raft_consensus.cc:686] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P 2d027e7df09047df8520a2776a891718 [term 1 LEADER]: Becoming Leader. State: Replica: 2d027e7df09047df8520a2776a891718, State: Running, Role: LEADER
I20260620 18:41:46.693580 7657 consensus_queue.cc:237] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P 2d027e7df09047df8520a2776a891718 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d027e7df09047df8520a2776a891718" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35637 } } peers { permanent_uuid: "cd6a34c3397b44ce85248a5a9c85f066" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33533 } } peers { permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 } }
I20260620 18:41:46.695577 7669 tablet_replica.cc:439] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P 2d027e7df09047df8520a2776a891718: TxnStatusTablet state changed. Reason: New leader 2d027e7df09047df8520a2776a891718. Latest consensus state: current_term: 1 leader_uuid: "2d027e7df09047df8520a2776a891718" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d027e7df09047df8520a2776a891718" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35637 } } peers { permanent_uuid: "cd6a34c3397b44ce85248a5a9c85f066" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33533 } } peers { permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 } } }
I20260620 18:41:46.695928 7447 raft_consensus.cc:2454] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P cd6a34c3397b44ce85248a5a9c85f066 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d027e7df09047df8520a2776a891718 in term 1.
I20260620 18:41:46.695990 7669 tablet_replica.cc:442] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P 2d027e7df09047df8520a2776a891718: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:41:46.698040 7144 catalog_manager.cc:5495] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P 2d027e7df09047df8520a2776a891718 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d027e7df09047df8520a2776a891718 (127.5.225.129). New cstate: current_term: 1 leader_uuid: "2d027e7df09047df8520a2776a891718" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d027e7df09047df8520a2776a891718" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35637 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd6a34c3397b44ce85248a5a9c85f066" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 } health_report { overall_health: UNKNOWN } } }
I20260620 18:41:46.706624 7695 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 18:41:46.706866 7695 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 18:41:46.721879 7645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:46.716106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59158 (local address 127.5.225.129:35637)
0620 18:41:46.716297 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:46.716302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:46.716367 (+ 65us) server_negotiation.cc:407] Connection header received
0620 18:41:46.716623 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:46.716627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:46.716697 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:46.716772 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:46.717238 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.718072 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.720096 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.720548 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.720646 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:46.721209 (+ 563us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:46.721312 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:46.721390 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:46.721594 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 18:41:46.721657 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:41:46.722030 7690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:46.715999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:35637 (local address 127.0.0.1:59158)
0620 18:41:46.716267 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:46.716290 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:46.716452 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:46.716835 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:46.716840 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:46.716855 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:41:46.717066 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:46.717079 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.718236 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.718242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:46.719924 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:46.719947 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.720709 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.720717 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:46.720952 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:46.721026 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:41:46.721426 (+ 400us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:41:46.721448 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 18:41:46.721546 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 18:41:46.721703 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 18:41:46.725890 7447 raft_consensus.cc:1261] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P cd6a34c3397b44ce85248a5a9c85f066 [term 1 FOLLOWER]: Refusing update from remote peer 2d027e7df09047df8520a2776a891718: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:41:46.725914 7586 raft_consensus.cc:1261] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P c20fc465cf9f4ea7b5cddf62855dc2bd [term 1 FOLLOWER]: Refusing update from remote peer 2d027e7df09047df8520a2776a891718: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:41:46.726550 7669 consensus_queue.cc:1035] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P 2d027e7df09047df8520a2776a891718 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:41:46.726855 7657 consensus_queue.cc:1035] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P 2d027e7df09047df8520a2776a891718 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd6a34c3397b44ce85248a5a9c85f066" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:41:46.730420 7656 tablet_replica.cc:439] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P c20fc465cf9f4ea7b5cddf62855dc2bd: TxnStatusTablet state changed. Reason: New leader 2d027e7df09047df8520a2776a891718. Latest consensus state: current_term: 1 leader_uuid: "2d027e7df09047df8520a2776a891718" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d027e7df09047df8520a2776a891718" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35637 } } peers { permanent_uuid: "cd6a34c3397b44ce85248a5a9c85f066" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33533 } } peers { permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 } } }
I20260620 18:41:46.730721 7656 tablet_replica.cc:442] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P c20fc465cf9f4ea7b5cddf62855dc2bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:41:46.732538 7661 tablet_replica.cc:439] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P 2d027e7df09047df8520a2776a891718: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2d027e7df09047df8520a2776a891718" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d027e7df09047df8520a2776a891718" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35637 } } peers { permanent_uuid: "cd6a34c3397b44ce85248a5a9c85f066" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33533 } } peers { permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 } } }
I20260620 18:41:46.732841 7661 tablet_replica.cc:442] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P 2d027e7df09047df8520a2776a891718: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:41:46.735296 7665 mvcc.cc:205] Tried to move back new op lower bound from 7298993793943146496 to 7298993793820647424. Current Snapshot: MvccSnapshot[applied={T|T < 7298993793943146496 or (T in {7298993793943146496})}]
I20260620 18:41:46.735399 7664 mvcc.cc:205] Tried to move back new op lower bound from 7298993793943146496 to 7298993793820647424. Current Snapshot: MvccSnapshot[applied={T|T < 7298993793943146496 or (T in {7298993793943146496})}]
I20260620 18:41:46.735728 7656 tablet_replica.cc:439] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P c20fc465cf9f4ea7b5cddf62855dc2bd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2d027e7df09047df8520a2776a891718" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d027e7df09047df8520a2776a891718" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35637 } } peers { permanent_uuid: "cd6a34c3397b44ce85248a5a9c85f066" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33533 } } peers { permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 } } }
I20260620 18:41:46.735934 7656 tablet_replica.cc:442] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P c20fc465cf9f4ea7b5cddf62855dc2bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:41:46.741680 7658 tablet_replica.cc:439] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P cd6a34c3397b44ce85248a5a9c85f066: TxnStatusTablet state changed. Reason: New leader 2d027e7df09047df8520a2776a891718. Latest consensus state: current_term: 1 leader_uuid: "2d027e7df09047df8520a2776a891718" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d027e7df09047df8520a2776a891718" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35637 } } peers { permanent_uuid: "cd6a34c3397b44ce85248a5a9c85f066" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33533 } } peers { permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 } } }
I20260620 18:41:46.742061 7658 tablet_replica.cc:442] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P cd6a34c3397b44ce85248a5a9c85f066: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:41:46.743566 7661 tablet_replica.cc:439] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P 2d027e7df09047df8520a2776a891718: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2d027e7df09047df8520a2776a891718" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d027e7df09047df8520a2776a891718" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35637 } } peers { permanent_uuid: "cd6a34c3397b44ce85248a5a9c85f066" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33533 } } peers { permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 } } }
I20260620 18:41:46.743877 7661 tablet_replica.cc:442] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P 2d027e7df09047df8520a2776a891718: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:41:46.745199 7666 mvcc.cc:205] Tried to move back new op lower bound from 7298993793943146496 to 7298993793820647424. Current Snapshot: MvccSnapshot[applied={T|T < 7298993793943146496}]
I20260620 18:41:46.746244 7658 tablet_replica.cc:439] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P cd6a34c3397b44ce85248a5a9c85f066: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2d027e7df09047df8520a2776a891718" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d027e7df09047df8520a2776a891718" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35637 } } peers { permanent_uuid: "cd6a34c3397b44ce85248a5a9c85f066" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33533 } } peers { permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 } } }
I20260620 18:41:46.746507 7658 tablet_replica.cc:442] T 743cdbaf7d7d4eeb88ed6ffba18ac9d9 P cd6a34c3397b44ce85248a5a9c85f066: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:41:46.750314 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:46.740846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57348 (local address 127.5.225.130:33533)
0620 18:41:46.742591 (+ 1745us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:46.742598 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:46.742616 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:41:46.742712 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:46.742719 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:46.742826 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:46.743417 (+ 591us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:46.744602 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.745793 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.747752 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.748362 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.748454 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:46.749062 (+ 608us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:46.749626 (+ 564us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:46.749738 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:46.749884 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 18:41:46.749946 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1615,"thread_start_us":1476,"threads_started":1}
I20260620 18:41:46.766037 7702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:46.760080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:35637 (local address 127.5.225.130:60599)
0620 18:41:46.760694 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:46.760713 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:46.760817 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:46.761123 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:46.761126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:46.761134 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:41:46.761389 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:46.761418 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.762852 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.762856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:46.763914 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:46.763925 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.764943 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.764948 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:46.765070 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:46.765117 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:41:46.765567 (+ 450us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:41:46.765577 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 18:41:46.765653 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 18:41:46.765747 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":333,"threads_started":1}
I20260620 18:41:46.766062 7645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:46.760075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:60599 (local address 127.5.225.129:35637)
0620 18:41:46.760303 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:46.760310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:46.760790 (+ 480us) server_negotiation.cc:407] Connection header received
0620 18:41:46.760930 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:46.760935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:46.761014 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:46.761106 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:46.761611 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.762721 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.764050 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.764780 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.764908 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:46.765277 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:46.765450 (+ 173us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:46.765539 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:46.765695 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 18:41:46.765772 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:41:46.775681 7585 raft_consensus.cc:1261] T fddde2e47e5246c2933a1ffda90d6713 P c20fc465cf9f4ea7b5cddf62855dc2bd [term 1 FOLLOWER]: Refusing update from remote peer cd6a34c3397b44ce85248a5a9c85f066: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:41:46.776003 7303 raft_consensus.cc:1261] T fddde2e47e5246c2933a1ffda90d6713 P 2d027e7df09047df8520a2776a891718 [term 1 FOLLOWER]: Refusing update from remote peer cd6a34c3397b44ce85248a5a9c85f066: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:41:46.776734 7682 consensus_queue.cc:1035] T fddde2e47e5246c2933a1ffda90d6713 P cd6a34c3397b44ce85248a5a9c85f066 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d027e7df09047df8520a2776a891718" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:41:46.777091 7658 consensus_queue.cc:1035] T fddde2e47e5246c2933a1ffda90d6713 P cd6a34c3397b44ce85248a5a9c85f066 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c20fc465cf9f4ea7b5cddf62855dc2bd" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:41:46.828342 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:46.821189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:54615 (local address 127.5.225.130:33533)
0620 18:41:46.821377 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:46.821384 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:46.822346 (+ 962us) server_negotiation.cc:407] Connection header received
0620 18:41:46.822685 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:46.822691 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:46.822783 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:46.822883 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:46.823377 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.824545 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.826441 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.826983 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.827096 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:46.827608 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:46.827743 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:46.827837 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:46.827986 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 18:41:46.828067 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:41:46.828446 7706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:46.821089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.130:33533 (local address 127.5.225.129:54615)
0620 18:41:46.822247 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:46.822267 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:46.822524 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:46.822901 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:46.822905 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:46.822914 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:41:46.823226 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:46.823234 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.824691 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.824695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:46.825951 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:46.825963 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.827139 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.827146 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:46.827339 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:46.827421 (+ 82us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:41:46.827846 (+ 425us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:41:46.827855 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 18:41:46.827947 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 18:41:46.828075 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"thread_start_us":722,"threads_started":1}
I20260620 18:41:46.871753 7146 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:41:46.872027 7146 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:41:46.875654 7146 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f63f8e6e7e064537a0e37780fdbab743: Sending DeleteTablet for 3 replicas of tablet fddde2e47e5246c2933a1ffda90d6713
I20260620 18:41:46.876484 7282 tablet_service.cc:1514] Processing DeleteTablet for tablet fddde2e47e5246c2933a1ffda90d6713 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:41:46 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:58966
I20260620 18:41:46.876915 7563 tablet_service.cc:1514] Processing DeleteTablet for tablet fddde2e47e5246c2933a1ffda90d6713 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:41:46 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:59066
I20260620 18:41:46.877702 7423 tablet_service.cc:1514] Processing DeleteTablet for tablet fddde2e47e5246c2933a1ffda90d6713 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:41:46 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:51856
I20260620 18:41:46.877981 7675 tablet_replica.cc:330] T fddde2e47e5246c2933a1ffda90d6713 P cd6a34c3397b44ce85248a5a9c85f066: stopping tablet replica
I20260620 18:41:46.878242 7675 raft_consensus.cc:2229] T fddde2e47e5246c2933a1ffda90d6713 P cd6a34c3397b44ce85248a5a9c85f066 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:41:46.878814 7675 raft_consensus.cc:2258] T fddde2e47e5246c2933a1ffda90d6713 P cd6a34c3397b44ce85248a5a9c85f066 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:41:46.880016 7675 ts_tablet_manager.cc:1905] T fddde2e47e5246c2933a1ffda90d6713 P cd6a34c3397b44ce85248a5a9c85f066: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:41:46.889580 7675 ts_tablet_manager.cc:1918] T fddde2e47e5246c2933a1ffda90d6713 P cd6a34c3397b44ce85248a5a9c85f066: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:41:46.892532 7675 log.cc:1196] T fddde2e47e5246c2933a1ffda90d6713 P cd6a34c3397b44ce85248a5a9c85f066: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-1/wal/wals/fddde2e47e5246c2933a1ffda90d6713
I20260620 18:41:46.893697 7675 ts_tablet_manager.cc:1939] T fddde2e47e5246c2933a1ffda90d6713 P cd6a34c3397b44ce85248a5a9c85f066: Deleting consensus metadata
I20260620 18:41:46.895473 7132 catalog_manager.cc:4841] TS cd6a34c3397b44ce85248a5a9c85f066 (127.5.225.130:33533): tablet fddde2e47e5246c2933a1ffda90d6713 (table test-table [id=cd492c4e6c6a40bbaa55dabf29c9560e]) successfully deleted
I20260620 18:41:46.900985 7710 tablet_replica.cc:330] T fddde2e47e5246c2933a1ffda90d6713 P 2d027e7df09047df8520a2776a891718: stopping tablet replica
I20260620 18:41:46.901217 7711 tablet_replica.cc:330] T fddde2e47e5246c2933a1ffda90d6713 P c20fc465cf9f4ea7b5cddf62855dc2bd: stopping tablet replica
I20260620 18:41:46.901273 7710 raft_consensus.cc:2229] T fddde2e47e5246c2933a1ffda90d6713 P 2d027e7df09047df8520a2776a891718 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:41:46.901419 7711 raft_consensus.cc:2229] T fddde2e47e5246c2933a1ffda90d6713 P c20fc465cf9f4ea7b5cddf62855dc2bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:41:46.901669 7711 raft_consensus.cc:2258] T fddde2e47e5246c2933a1ffda90d6713 P c20fc465cf9f4ea7b5cddf62855dc2bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:41:46.901820 7710 raft_consensus.cc:2258] T fddde2e47e5246c2933a1ffda90d6713 P 2d027e7df09047df8520a2776a891718 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:41:46.903187 7710 ts_tablet_manager.cc:1905] T fddde2e47e5246c2933a1ffda90d6713 P 2d027e7df09047df8520a2776a891718: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:41:46.903234 7711 ts_tablet_manager.cc:1905] T fddde2e47e5246c2933a1ffda90d6713 P c20fc465cf9f4ea7b5cddf62855dc2bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:41:46.910341 7711 ts_tablet_manager.cc:1918] T fddde2e47e5246c2933a1ffda90d6713 P c20fc465cf9f4ea7b5cddf62855dc2bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:41:46.910472 7711 log.cc:1196] T fddde2e47e5246c2933a1ffda90d6713 P c20fc465cf9f4ea7b5cddf62855dc2bd: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-2/wal/wals/fddde2e47e5246c2933a1ffda90d6713
I20260620 18:41:46.910774 7710 ts_tablet_manager.cc:1918] T fddde2e47e5246c2933a1ffda90d6713 P 2d027e7df09047df8520a2776a891718: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:41:46.910876 7711 ts_tablet_manager.cc:1939] T fddde2e47e5246c2933a1ffda90d6713 P c20fc465cf9f4ea7b5cddf62855dc2bd: Deleting consensus metadata
I20260620 18:41:46.911001 7710 log.cc:1196] T fddde2e47e5246c2933a1ffda90d6713 P 2d027e7df09047df8520a2776a891718: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980894474392-6022-0/minicluster-data/ts-0/wal/wals/fddde2e47e5246c2933a1ffda90d6713
I20260620 18:41:46.911527 7710 ts_tablet_manager.cc:1939] T fddde2e47e5246c2933a1ffda90d6713 P 2d027e7df09047df8520a2776a891718: Deleting consensus metadata
I20260620 18:41:46.912009 7130 catalog_manager.cc:4841] TS c20fc465cf9f4ea7b5cddf62855dc2bd (127.5.225.131:46683): tablet fddde2e47e5246c2933a1ffda90d6713 (table test-table [id=cd492c4e6c6a40bbaa55dabf29c9560e]) successfully deleted
I20260620 18:41:46.912703 7130 catalog_manager.cc:4841] TS 2d027e7df09047df8520a2776a891718 (127.5.225.129:35637): tablet fddde2e47e5246c2933a1ffda90d6713 (table test-table [id=cd492c4e6c6a40bbaa55dabf29c9560e]) successfully deleted
Jun 20 18:41:46 dist-test-slave-k6lg krb5kdc[7101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980906, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.225.129@KRBTEST.COM
I20260620 18:41:46.933652 7645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:46.913137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59172 (local address 127.5.225.129:35637)
0620 18:41:46.913344 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:46.913351 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:46.914377 (+ 1026us) server_negotiation.cc:407] Connection header received
0620 18:41:46.914636 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:46.914642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:46.914729 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:46.914832 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:46.916699 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.917889 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.919888 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.920594 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.920724 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:46.925986 (+ 5262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:46.926005 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:46.926026 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:46.926072 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:46.930437 (+ 4365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:46.931305 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:46.931315 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:46.931320 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:46.931482 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:46.931912 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:46.931917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:46.931922 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:46.932384 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:46.932721 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:46.933013 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 18:41:46.933266 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 18:41:46.934474 7348 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59172
I20260620 18:41:46.961237 7689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:46.939353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38442 (local address 127.5.225.190:44115)
0620 18:41:46.939586 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:46.939590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:46.942669 (+ 3079us) server_negotiation.cc:407] Connection header received
0620 18:41:46.942829 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:46.942835 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:46.942922 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:46.943027 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:46.944673 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.945825 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.947752 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.948418 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.948548 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:46.951393 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:46.951400 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:46.951405 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:46.951446 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:46.955013 (+ 3567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:46.955808 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:46.955814 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:46.955818 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:46.955958 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:46.956351 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:46.956359 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:46.956364 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:46.956706 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:46.956948 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:46.957285 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 18:41:46.957507 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
W20260620 18:41:46.962136 7146 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38442
I20260620 18:41:46.964262 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 7217
I20260620 18:41:46.982134 7645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:46.965580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59182 (local address 127.5.225.129:35637)
0620 18:41:46.965763 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:46.965769 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:46.966250 (+ 481us) server_negotiation.cc:407] Connection header received
0620 18:41:46.966496 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:46.966505 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:46.966594 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:46.966663 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:46.968328 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.969604 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.971416 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:46.972130 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:46.972233 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:46.975148 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:46.975158 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:46.975163 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:46.975200 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:46.978902 (+ 3702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:46.979761 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:46.979767 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:46.979772 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:46.979958 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:46.980386 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:46.980392 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:46.980395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:46.980701 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:46.980963 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:46.981502 (+ 539us) server_negotiation.cc:299] Negotiation successful
0620 18:41:46.981731 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 18:41:46.983364 7348 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59182
W20260620 18:41:46.983898 6022 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 7217: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:41:46.984100 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 7217
I20260620 18:41:47.013134 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 7357
Jun 20 18:41:47 dist-test-slave-k6lg krb5kdc[7101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980906, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.225.130@KRBTEST.COM
I20260620 18:41:47.030467 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:47.013924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57364 (local address 127.5.225.130:33533)
0620 18:41:47.014199 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:47.014206 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:47.014229 (+ 23us) server_negotiation.cc:407] Connection header received
0620 18:41:47.014394 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:47.014397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:47.014476 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:47.014563 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:47.016097 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:47.016859 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:47.018685 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:47.019117 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:47.019208 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:47.024150 (+ 4942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:47.024168 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:47.024184 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:47.024223 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:47.027412 (+ 3189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:47.028134 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:47.028142 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:47.028145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:47.028285 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:47.028755 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:47.028759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:47.028761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:47.029182 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:47.029521 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:47.029809 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 18:41:47.030047 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
W20260620 18:41:47.031107 7487 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57364
W20260620 18:41:47.031749 6022 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 7357: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:41:47.031922 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 7357
I20260620 18:41:47.050109 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 7496
Jun 20 18:41:47 dist-test-slave-k6lg krb5kdc[7101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980906, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.225.131@KRBTEST.COM
I20260620 18:41:47.064675 7649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:47.050981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50700 (local address 127.5.225.131:46683)
0620 18:41:47.051167 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:47.051171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:47.051193 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:41:47.051332 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:47.051336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:47.051405 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:47.051497 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:47.052671 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:47.053427 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:47.054726 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:47.055181 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:47.055276 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:47.059287 (+ 4011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:47.059301 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:47.059316 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:47.059351 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:47.062145 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:47.062884 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:47.062892 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:47.062894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:47.063033 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:47.063406 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:47.063409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:47.063411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:47.063770 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:47.063986 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:47.064271 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 18:41:47.064428 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 18:41:47.065338 7626 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50700
W20260620 18:41:47.066042 6022 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 7496: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:41:47.066210 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 7496
I20260620 18:41:47.082988 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 7114
I20260620 18:41:47.084052 7186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:47.199718 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 7114
2026-06-20T18:41:47Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4588 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:41:47 dist-test-slave-k6lg krb5kdc[7720](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:41:47 dist-test-slave-k6lg krb5kdc[7720](info): set up 2 sockets
Jun 20 18:41:47 dist-test-slave-k6lg krb5kdc[7720](info): commencing operation
krb5kdc: starting...
W20260620 18:41:48.286510 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:41:48 dist-test-slave-k6lg krb5kdc[7720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:41:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:41:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
I20260620 18:41:48.463546 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:43537
--webserver_interface=127.5.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:37573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:43537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:48.780895 7733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:48.781208 7733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:48.781368 7733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:48.788713 7733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:41:48.788812 7733 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 18:41:48.788878 7733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:48.788975 7733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:48.789068 7733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:41:48.789160 7733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:41:48.799664 7733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:37573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:43537
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:43537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:41:48.801007 7733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:48.802425 7733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:48.814284 7738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:48.814240 7739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:48.814307 7741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:48.815423 7733 server_base.cc:1034] running on GCE node
I20260620 18:41:48.816107 7733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:48.817387 7733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:48.818635 7733 hybrid_clock.cc:648] HybridClock initialized: now 1781980908818549 us; error 53 us; skew 500 ppm
I20260620 18:41:48.818924 7733 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:48 dist-test-slave-k6lg krb5kdc[7720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980908, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:48.827468 7733 init.cc:377] Logged in from keytab as kudu/127.5.225.190@KRBTEST.COM (short username kudu)
I20260620 18:41:48.829025 7733 webserver.cc:460] Webserver started at http://127.5.225.190:33479/ using document root <none> and password file <none>
I20260620 18:41:48.829718 7733 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:48.829806 7733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:48.830080 7733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:48.833132 7733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "a6100e6394dd4f48838ad1e248e2f501"
format_stamp: "Formatted at 2026-06-20 18:41:48 on dist-test-slave-k6lg"
server_key: "9c8f3a39dadaad21d4ac82a3a0d1591a"
server_key_iv: "5f7ae77135e22f72053fe2facc188609"
server_key_version: "encryptionkey@0"
I20260620 18:41:48.833846 7733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "a6100e6394dd4f48838ad1e248e2f501"
format_stamp: "Formatted at 2026-06-20 18:41:48 on dist-test-slave-k6lg"
server_key: "9c8f3a39dadaad21d4ac82a3a0d1591a"
server_key_iv: "5f7ae77135e22f72053fe2facc188609"
server_key_version: "encryptionkey@0"
I20260620 18:41:48.839210 7733 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 18:41:48.843267 7748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:48.844918 7733 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:41:48.845129 7733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "a6100e6394dd4f48838ad1e248e2f501"
format_stamp: "Formatted at 2026-06-20 18:41:48 on dist-test-slave-k6lg"
server_key: "9c8f3a39dadaad21d4ac82a3a0d1591a"
server_key_iv: "5f7ae77135e22f72053fe2facc188609"
server_key_version: "encryptionkey@0"
I20260620 18:41:48.845378 7733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:48.860219 7733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:48.864874 7733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:48.865175 7733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:48.887251 7733 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:43537
I20260620 18:41:48.887277 7809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:43537 every 8 connection(s)
I20260620 18:41:48.888711 7733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:41:48.890538 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 7733
I20260620 18:41:48.890754 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:41:48.891398 6022 external_mini_cluster.cc:1428] Setting key b6a51013f0f0870bfe86a8898afb7330
I20260620 18:41:48.892976 7810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 18:41:48 dist-test-slave-k6lg krb5kdc[7720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:41:48.907142 7810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6100e6394dd4f48838ad1e248e2f501: Bootstrap starting.
I20260620 18:41:48.911386 7810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6100e6394dd4f48838ad1e248e2f501: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:48.912669 7810 log.cc:824] T 00000000000000000000000000000000 P a6100e6394dd4f48838ad1e248e2f501: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:48.915586 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:48.893450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47616 (local address 127.5.225.190:43537)
0620 18:41:48.894252 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:48.894263 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:48.894304 (+ 41us) server_negotiation.cc:407] Connection header received
0620 18:41:48.895846 (+ 1542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:48.895866 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:48.896255 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:48.896588 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:48.898409 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:48.899780 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:48.901505 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:48.902220 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:48.902361 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:48.907137 (+ 4776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:48.907148 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:48.907170 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:48.907222 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:48.911012 (+ 3790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:48.911814 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:48.911827 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:48.911837 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:48.912032 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:48.912383 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:48.912387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:48.912390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:48.913095 (+ 705us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:48.913484 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:48.913871 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 18:41:48.914320 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":355,"threads_started":1}
I20260620 18:41:48.916211 7810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6100e6394dd4f48838ad1e248e2f501: No bootstrap required, opened a new log
I20260620 18:41:48.923735 7810 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6100e6394dd4f48838ad1e248e2f501 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6100e6394dd4f48838ad1e248e2f501" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 43537 } }
I20260620 18:41:48.924149 7810 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6100e6394dd4f48838ad1e248e2f501 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:48.924222 7810 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6100e6394dd4f48838ad1e248e2f501 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6100e6394dd4f48838ad1e248e2f501, State: Initialized, Role: FOLLOWER
I20260620 18:41:48.924793 7810 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6100e6394dd4f48838ad1e248e2f501 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6100e6394dd4f48838ad1e248e2f501" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 43537 } }
I20260620 18:41:48.924988 7810 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6100e6394dd4f48838ad1e248e2f501 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:41:48.925079 7810 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6100e6394dd4f48838ad1e248e2f501 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:41:48.925200 7810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6100e6394dd4f48838ad1e248e2f501 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:48.927397 7810 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6100e6394dd4f48838ad1e248e2f501 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6100e6394dd4f48838ad1e248e2f501" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 43537 } }
I20260620 18:41:48.927850 7810 leader_election.cc:302] T 00000000000000000000000000000000 P a6100e6394dd4f48838ad1e248e2f501 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6100e6394dd4f48838ad1e248e2f501; no voters:
I20260620 18:41:48.928392 7810 leader_election.cc:288] T 00000000000000000000000000000000 P a6100e6394dd4f48838ad1e248e2f501 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:41:48.928593 7815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6100e6394dd4f48838ad1e248e2f501 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:41:48.929746 7815 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6100e6394dd4f48838ad1e248e2f501 [term 1 LEADER]: Becoming Leader. State: Replica: a6100e6394dd4f48838ad1e248e2f501, State: Running, Role: LEADER
I20260620 18:41:48.930315 7815 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6100e6394dd4f48838ad1e248e2f501 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6100e6394dd4f48838ad1e248e2f501" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 43537 } }
I20260620 18:41:48.930711 7810 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6100e6394dd4f48838ad1e248e2f501 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:41:48.933183 7817 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6100e6394dd4f48838ad1e248e2f501 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6100e6394dd4f48838ad1e248e2f501. Latest consensus state: current_term: 1 leader_uuid: "a6100e6394dd4f48838ad1e248e2f501" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6100e6394dd4f48838ad1e248e2f501" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 43537 } } }
I20260620 18:41:48.933619 7817 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6100e6394dd4f48838ad1e248e2f501 [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:48.933832 7816 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6100e6394dd4f48838ad1e248e2f501 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6100e6394dd4f48838ad1e248e2f501" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6100e6394dd4f48838ad1e248e2f501" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 43537 } } }
I20260620 18:41:48.934118 7816 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6100e6394dd4f48838ad1e248e2f501 [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:48.935822 7821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:41:48.941146 7821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:41:48.950147 7821 catalog_manager.cc:1347] Generated new cluster ID: 62cac70a82c44ba69c8a8ea0682781fc
I20260620 18:41:48.950282 7821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:41:48.961858 7821 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:41:48.963146 7821 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:41:48.971745 7821 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6100e6394dd4f48838ad1e248e2f501: Generated new TSK 0
I20260620 18:41:48.972764 7821 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab.
Entry for principal kudu/127.5.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab.
I20260620 18:41:49.034384 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.129:0
--local_ip_for_outbound_sockets=127.5.225.129
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:43537
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:37573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:49.358724 7836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:49.358989 7836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:49.359087 7836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:49.366061 7836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:49.366222 7836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:49.366334 7836 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 18:41:49.366389 7836 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:41:49.366451 7836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.129
I20260620 18:41:49.376233 7836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:37573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.225.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.225.190:43537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.129
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:41:49.377457 7836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:49.378760 7836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:49.392333 7841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:49.392453 7844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:49.392499 7842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:49.393683 7836 server_base.cc:1034] running on GCE node
I20260620 18:41:49.394505 7836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:49.398829 7836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:49.400147 7836 hybrid_clock.cc:648] HybridClock initialized: now 1781980909400038 us; error 100 us; skew 500 ppm
I20260620 18:41:49.400621 7836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:49 dist-test-slave-k6lg krb5kdc[7720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980909, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:49.405335 7836 init.cc:377] Logged in from keytab as kudu/127.5.225.129@KRBTEST.COM (short username kudu)
I20260620 18:41:49.407331 7836 webserver.cc:460] Webserver started at http://127.5.225.129:39243/ using document root <none> and password file <none>
I20260620 18:41:49.408344 7836 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:49.408499 7836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:49.408825 7836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:49.412052 7836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-0/data/instance:
uuid: "b354f0c3d7444c3281180585aeed4c5b"
format_stamp: "Formatted at 2026-06-20 18:41:49 on dist-test-slave-k6lg"
server_key: "408401ca4d2168e795ad8211187f6f0e"
server_key_iv: "16f502bf27e9faac781c7c5a971452f5"
server_key_version: "encryptionkey@0"
I20260620 18:41:49.412909 7836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance:
uuid: "b354f0c3d7444c3281180585aeed4c5b"
format_stamp: "Formatted at 2026-06-20 18:41:49 on dist-test-slave-k6lg"
server_key: "408401ca4d2168e795ad8211187f6f0e"
server_key_iv: "16f502bf27e9faac781c7c5a971452f5"
server_key_version: "encryptionkey@0"
I20260620 18:41:49.418754 7836 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 18:41:49.422430 7851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:49.423849 7836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:41:49.424046 7836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-0/wal
uuid: "b354f0c3d7444c3281180585aeed4c5b"
format_stamp: "Formatted at 2026-06-20 18:41:49 on dist-test-slave-k6lg"
server_key: "408401ca4d2168e795ad8211187f6f0e"
server_key_iv: "16f502bf27e9faac781c7c5a971452f5"
server_key_version: "encryptionkey@0"
I20260620 18:41:49.424259 7836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:49.442731 7836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:49.446779 7836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:49.447077 7836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:49.450059 7836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:49.451989 7836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:49.452152 7836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:49.452265 7836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:41:49 dist-test-slave-k6lg krb5kdc[7720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980909, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:41:49.477039 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:49.450339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:38601 (local address 127.5.225.190:43537)
0620 18:41:49.450788 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:49.450793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:49.451925 (+ 1132us) server_negotiation.cc:407] Connection header received
0620 18:41:49.454325 (+ 2400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:49.454332 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:49.454439 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:49.454568 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:49.456211 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:49.457224 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:49.460194 (+ 2970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:49.460847 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:49.461016 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:49.467577 (+ 6561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:49.467589 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:49.467597 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:49.467647 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:49.472355 (+ 4708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:49.473586 (+ 1231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:49.473593 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:49.473597 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:49.473740 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:49.474361 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:49.474369 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:49.474377 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:49.474680 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:49.474849 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:49.476052 (+ 1203us) server_negotiation.cc:299] Negotiation successful
0620 18:41:49.476389 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":262,"threads_started":1}
I20260620 18:41:49.477710 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:49.450867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:43537 (local address 127.5.225.129:38601)
0620 18:41:49.451733 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:49.451775 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:49.453957 (+ 2182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:49.454852 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:49.454860 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:49.455394 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:49.456023 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:49.456042 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:49.457416 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:49.457421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:49.459977 (+ 2556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:49.460000 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:49.461264 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:49.461273 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:49.461739 (+ 466us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:49.462681 (+ 942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:49.462731 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:49.467169 (+ 4438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:49.472540 (+ 5371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:49.472552 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:49.472580 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:49.473433 (+ 853us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:49.473898 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:49.473904 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:49.473908 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:49.474199 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:49.474842 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:49.474851 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:49.475458 (+ 607us) client_negotiation.cc:769] Sending connection context
0620 18:41:49.475868 (+ 410us) client_negotiation.cc:240] Negotiation successful
0620 18:41:49.476353 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":329,"threads_started":1}
I20260620 18:41:49.499706 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:49.484259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:33493 (local address 127.5.225.190:43537)
0620 18:41:49.484506 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:49.484515 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:49.485054 (+ 539us) server_negotiation.cc:407] Connection header received
0620 18:41:49.485163 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:49.485173 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:49.485283 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:49.485428 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:49.486467 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:49.488061 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:49.489426 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:49.490366 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:49.490533 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:49.493408 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:49.493416 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:49.493424 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:49.493461 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:49.496564 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:49.497447 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:49.497455 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:49.497461 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:49.497643 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:49.498191 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:49.498198 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:49.498203 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:49.498549 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:49.498701 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:49.499115 (+ 414us) server_negotiation.cc:299] Negotiation successful
0620 18:41:49.499336 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 18:41:49.499814 7931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:49.484236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:43537 (local address 127.5.225.129:33493)
0620 18:41:49.484921 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:49.484939 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:49.485057 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:49.485458 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:49.485463 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:49.485945 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:49.486317 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:49.486327 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:49.488242 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:49.488245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:49.489273 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:49.489284 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:49.490600 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:49.490606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:49.490726 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:49.491439 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:49.491471 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:49.493188 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:49.496700 (+ 3512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:49.496709 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:49.496713 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:49.497322 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:49.497780 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:49.497789 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:49.497795 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:49.498028 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:49.498673 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:49.498678 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:49.498871 (+ 193us) client_negotiation.cc:769] Sending connection context
0620 18:41:49.499011 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 18:41:49.499337 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":367,"threads_started":1}
I20260620 18:41:49.508033 7836 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.129:37425
I20260620 18:41:49.508067 7971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.129:37425 every 8 connection(s)
I20260620 18:41:49.509328 7836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
I20260620 18:41:49.511457 7972 heartbeater.cc:347] Connected to a master server at 127.5.225.190:43537
I20260620 18:41:49.511682 7972 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:49.512384 7972 heartbeater.cc:511] Master 127.5.225.190:43537 requested a full tablet report, sending...
I20260620 18:41:49.512880 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 7836
I20260620 18:41:49.513087 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance
I20260620 18:41:49.513800 6022 external_mini_cluster.cc:1428] Setting key 6aae2be0670b42cdbf87a83b32554524
I20260620 18:41:49.514835 7765 ts_manager.cc:194] Registered new tserver with Master: b354f0c3d7444c3281180585aeed4c5b (127.5.225.129:37425)
I20260620 18:41:49.516659 7765 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.129@KRBTEST.COM'} at 127.5.225.129:38601
WARNING: no policy specified for kudu/127.5.225.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab.
Entry for principal kudu/127.5.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab.
I20260620 18:41:49.554563 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.130:0
--local_ip_for_outbound_sockets=127.5.225.130
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:43537
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:37573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:49.867653 7976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:49.867921 7976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:49.868096 7976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:49.875202 7976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:49.875303 7976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:49.875484 7976 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 18:41:49.875572 7976 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:41:49.875711 7976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.130
I20260620 18:41:49.885313 7976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:37573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.225.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.225.190:43537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.130
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:41:49.886569 7976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:49.887900 7976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:49.901027 7984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:49.900987 7982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:49.901450 7981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:49.902115 7976 server_base.cc:1034] running on GCE node
I20260620 18:41:49.902758 7976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:49.903752 7976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:49.904951 7976 hybrid_clock.cc:648] HybridClock initialized: now 1781980909904913 us; error 30 us; skew 500 ppm
I20260620 18:41:49.905254 7976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:49 dist-test-slave-k6lg krb5kdc[7720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980909, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:49.908387 7976 init.cc:377] Logged in from keytab as kudu/127.5.225.130@KRBTEST.COM (short username kudu)
I20260620 18:41:49.909730 7976 webserver.cc:460] Webserver started at http://127.5.225.130:37103/ using document root <none> and password file <none>
I20260620 18:41:49.910467 7976 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:49.910594 7976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:49.910912 7976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:49.913726 7976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-1/data/instance:
uuid: "aee666d8594f44b09b3d8f73784f3979"
format_stamp: "Formatted at 2026-06-20 18:41:49 on dist-test-slave-k6lg"
server_key: "6521c317dc3c6cccfe246eed1803830f"
server_key_iv: "f015ef52e8bfff1a6478865cf126d885"
server_key_version: "encryptionkey@0"
I20260620 18:41:49.914527 7976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance:
uuid: "aee666d8594f44b09b3d8f73784f3979"
format_stamp: "Formatted at 2026-06-20 18:41:49 on dist-test-slave-k6lg"
server_key: "6521c317dc3c6cccfe246eed1803830f"
server_key_iv: "f015ef52e8bfff1a6478865cf126d885"
server_key_version: "encryptionkey@0"
I20260620 18:41:49.919435 7976 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 18:41:49.922783 7991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:49.924081 7976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:41:49.924237 7976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-1/wal
uuid: "aee666d8594f44b09b3d8f73784f3979"
format_stamp: "Formatted at 2026-06-20 18:41:49 on dist-test-slave-k6lg"
server_key: "6521c317dc3c6cccfe246eed1803830f"
server_key_iv: "f015ef52e8bfff1a6478865cf126d885"
server_key_version: "encryptionkey@0"
I20260620 18:41:49.924412 7976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:49.939405 7976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:49.943666 7976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:49.943943 7976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:49.946403 7976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:49.946488 7976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:49.946640 7976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:49.946730 7976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:41:49 dist-test-slave-k6lg krb5kdc[7720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980909, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:41:49.970399 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:49.947167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:36809 (local address 127.5.225.190:43537)
0620 18:41:49.947359 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:49.947364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:49.948742 (+ 1378us) server_negotiation.cc:407] Connection header received
0620 18:41:49.951190 (+ 2448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:49.951199 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:49.951310 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:49.951441 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:49.953606 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:49.954999 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:49.957363 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:49.957945 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:49.958097 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:49.963294 (+ 5197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:49.963299 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:49.963302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:49.963332 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:49.966561 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:49.967500 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:49.967507 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:49.967513 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:49.967639 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:49.968264 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:49.968267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:49.968269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:49.968634 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:49.968793 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:49.969789 (+ 996us) server_negotiation.cc:299] Negotiation successful
0620 18:41:49.970017 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 18:41:49.971871 8000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:49.947719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:43537 (local address 127.5.225.130:36809)
0620 18:41:49.948559 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:49.948611 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:49.950768 (+ 2157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:49.951760 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:49.951770 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:49.952566 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:49.953360 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:49.953393 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:49.955187 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:49.955195 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:49.957143 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:49.957154 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:49.958261 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:49.958272 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:49.958618 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:49.959418 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:49.959458 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:49.963085 (+ 3627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:49.966700 (+ 3615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:49.966708 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:49.966724 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:49.967353 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:49.967750 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:49.967755 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:49.967759 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:49.968105 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:49.968863 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:49.968876 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:49.969373 (+ 497us) client_negotiation.cc:769] Sending connection context
0620 18:41:49.969694 (+ 321us) client_negotiation.cc:240] Negotiation successful
0620 18:41:49.970277 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":377,"threads_started":1}
I20260620 18:41:49.992491 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:49.977885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:35715 (local address 127.5.225.190:43537)
0620 18:41:49.978094 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:49.978103 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:49.978643 (+ 540us) server_negotiation.cc:407] Connection header received
0620 18:41:49.978751 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:49.978759 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:49.978857 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:49.978973 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:49.979809 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:49.981272 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:49.983168 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:49.983823 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:49.983986 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:49.985595 (+ 1609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:49.985602 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:49.985607 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:49.985645 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:49.989189 (+ 3544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:49.990011 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:49.990020 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:49.990024 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:49.990152 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:49.990609 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:49.990618 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:49.990623 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:49.991088 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:49.991230 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:49.991771 (+ 541us) server_negotiation.cc:299] Negotiation successful
0620 18:41:49.992101 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:41:49.992606 8082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:49.977867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:43537 (local address 127.5.225.130:35715)
0620 18:41:49.978544 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:49.978559 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:49.978658 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:49.979004 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:49.979007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:49.979384 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:49.979672 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:49.979680 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:49.981405 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:49.981412 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:49.983034 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:49.983046 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:49.984004 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:49.984009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:49.984123 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:49.984730 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:49.984746 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:49.985387 (+ 641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:49.989320 (+ 3933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:49.989325 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:49.989329 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:49.989871 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:49.990319 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:49.990325 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:49.990328 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:49.990456 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:49.991328 (+ 872us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:49.991333 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:49.991495 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 18:41:49.991608 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 18:41:49.991999 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":376,"threads_started":1}
I20260620 18:41:49.993364 7976 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.130:42107
I20260620 18:41:49.993371 8110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.130:42107 every 8 connection(s)
I20260620 18:41:49.994794 7976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
I20260620 18:41:49.997072 8111 heartbeater.cc:347] Connected to a master server at 127.5.225.190:43537
I20260620 18:41:49.997289 8111 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:49.997875 8111 heartbeater.cc:511] Master 127.5.225.190:43537 requested a full tablet report, sending...
I20260620 18:41:49.999266 7765 ts_manager.cc:194] Registered new tserver with Master: aee666d8594f44b09b3d8f73784f3979 (127.5.225.130:42107)
I20260620 18:41:50.000064 7765 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.130@KRBTEST.COM'} at 127.5.225.130:36809
I20260620 18:41:50.001829 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 7976
I20260620 18:41:50.001999 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance
I20260620 18:41:50.002575 6022 external_mini_cluster.cc:1428] Setting key 4f0be93df61646e6d40e44c73229a925
WARNING: no policy specified for kudu/127.5.225.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab.
Entry for principal kudu/127.5.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab.
I20260620 18:41:50.043654 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.131:0
--local_ip_for_outbound_sockets=127.5.225.131
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:43537
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:37573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:50.361828 8115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:50.362092 8115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:50.362277 8115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:50.369354 8115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:50.369477 8115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:50.369668 8115 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 18:41:50.369746 8115 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:41:50.369854 8115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.131
I20260620 18:41:50.379624 8115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:37573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.225.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.225.190:43537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.131
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:41:50.380880 8115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:50.382285 8115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:50.395403 8120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:50.395473 8123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:50.395440 8121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:50.396485 8115 server_base.cc:1034] running on GCE node
I20260620 18:41:50.397100 8115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:50.397982 8115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:50.399215 8115 hybrid_clock.cc:648] HybridClock initialized: now 1781980910399151 us; error 50 us; skew 500 ppm
I20260620 18:41:50.399526 8115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:50 dist-test-slave-k6lg krb5kdc[7720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980910, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:50.402691 8115 init.cc:377] Logged in from keytab as kudu/127.5.225.131@KRBTEST.COM (short username kudu)
I20260620 18:41:50.404142 8115 webserver.cc:460] Webserver started at http://127.5.225.131:40901/ using document root <none> and password file <none>
I20260620 18:41:50.404805 8115 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:50.404937 8115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:50.405251 8115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:50.408114 8115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-2/data/instance:
uuid: "f6141944f9d7444dae7291f7e93a0db5"
format_stamp: "Formatted at 2026-06-20 18:41:50 on dist-test-slave-k6lg"
server_key: "944b237a582ee659c04e90e438a7bbaf"
server_key_iv: "5491a4806d5f7a41d371d9e0c5658b59"
server_key_version: "encryptionkey@0"
I20260620 18:41:50.408890 8115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance:
uuid: "f6141944f9d7444dae7291f7e93a0db5"
format_stamp: "Formatted at 2026-06-20 18:41:50 on dist-test-slave-k6lg"
server_key: "944b237a582ee659c04e90e438a7bbaf"
server_key_iv: "5491a4806d5f7a41d371d9e0c5658b59"
server_key_version: "encryptionkey@0"
I20260620 18:41:50.414050 8115 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 18:41:50.417850 8130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:50.419180 8115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:41:50.419348 8115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-2/wal
uuid: "f6141944f9d7444dae7291f7e93a0db5"
format_stamp: "Formatted at 2026-06-20 18:41:50 on dist-test-slave-k6lg"
server_key: "944b237a582ee659c04e90e438a7bbaf"
server_key_iv: "5491a4806d5f7a41d371d9e0c5658b59"
server_key_version: "encryptionkey@0"
I20260620 18:41:50.419574 8115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:50.442056 8115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:50.446256 8115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:50.446544 8115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:50.449250 8115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:50.449422 8115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:50.449568 8115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:50.449674 8115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:41:50 dist-test-slave-k6lg krb5kdc[7720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980910, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:41:50.474789 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:50.450026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:43205 (local address 127.5.225.190:43537)
0620 18:41:50.450242 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:50.450248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:50.451672 (+ 1424us) server_negotiation.cc:407] Connection header received
0620 18:41:50.454673 (+ 3001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:50.454682 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:50.454771 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:50.454885 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:50.456988 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.457846 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.461274 (+ 3428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.461967 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.462134 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:50.466463 (+ 4329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:50.466477 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:50.466480 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:50.466517 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:50.470861 (+ 4344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:50.471663 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:50.471672 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:50.471679 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:50.471871 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:50.472386 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:50.472394 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:50.472399 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:50.472830 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:50.473035 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:50.473895 (+ 860us) server_negotiation.cc:299] Negotiation successful
0620 18:41:50.474239 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 18:41:50.475694 8139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:50.450614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:43537 (local address 127.5.225.131:43205)
0620 18:41:50.451472 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:50.451539 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:50.454298 (+ 2759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:50.455167 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:50.455182 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:50.455972 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:50.456728 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:50.456750 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.458041 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.458051 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:50.461011 (+ 2960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:50.461027 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.462458 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.462465 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:50.462647 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:50.463386 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:50.463416 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:50.466200 (+ 2784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:50.471023 (+ 4823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:50.471035 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:50.471060 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:50.471518 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:50.472044 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:50.472048 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:50.472051 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:50.472255 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:50.473014 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:50.473025 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:50.473377 (+ 352us) client_negotiation.cc:769] Sending connection context
0620 18:41:50.473692 (+ 315us) client_negotiation.cc:240] Negotiation successful
0620 18:41:50.474150 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":349,"threads_started":1}
I20260620 18:41:50.503396 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:50.486704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:54999 (local address 127.5.225.190:43537)
0620 18:41:50.486936 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:50.486944 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:50.487730 (+ 786us) server_negotiation.cc:407] Connection header received
0620 18:41:50.487888 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:50.487897 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:50.488009 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:50.488142 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:50.489123 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.490533 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.492696 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.493402 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.493562 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:50.496259 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:50.496263 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:50.496267 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:50.496296 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:50.500397 (+ 4101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:50.501084 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:50.501092 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:50.501098 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:50.501295 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:50.501705 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:50.501713 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:50.501719 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:50.502124 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:50.502317 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:50.502757 (+ 440us) server_negotiation.cc:299] Negotiation successful
0620 18:41:50.502971 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 18:41:50.503688 8218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:50.486824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:43537 (local address 127.5.225.131:54999)
0620 18:41:50.487605 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:50.487633 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:50.487755 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:50.488165 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:50.488169 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:50.488684 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:50.488981 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:50.488990 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.490732 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.490740 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:50.492553 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:50.492566 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.493660 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.493670 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:50.493908 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:50.494623 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:50.494659 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:50.496079 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:50.500557 (+ 4478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:50.500561 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:50.500563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:50.500963 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:50.501454 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:50.501458 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:50.501461 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:50.501593 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:50.502295 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:50.502299 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:50.502460 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 18:41:50.502653 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 18:41:50.503009 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":454,"threads_started":1}
I20260620 18:41:50.509532 8115 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.131:46387
I20260620 18:41:50.509565 8249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.131:46387 every 8 connection(s)
I20260620 18:41:50.510918 8115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
I20260620 18:41:50.512979 8250 heartbeater.cc:347] Connected to a master server at 127.5.225.190:43537
I20260620 18:41:50.513226 8250 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:50.513870 8250 heartbeater.cc:511] Master 127.5.225.190:43537 requested a full tablet report, sending...
I20260620 18:41:50.515178 7765 ts_manager.cc:194] Registered new tserver with Master: f6141944f9d7444dae7291f7e93a0db5 (127.5.225.131:46387)
I20260620 18:41:50.515969 7765 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.131@KRBTEST.COM'} at 127.5.225.131:43205
I20260620 18:41:50.519610 7972 heartbeater.cc:503] Master 127.5.225.190:43537 was elected leader, sending a full tablet report...
I20260620 18:41:50.520917 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 8115
I20260620 18:41:50.521129 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance
I20260620 18:41:50.521595 6022 external_mini_cluster.cc:1428] Setting key be6109507204cc73ea64bace128d9185
I20260620 18:41:50.524627 6022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 18:41:50 dist-test-slave-k6lg krb5kdc[7720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 18:41:50 dist-test-slave-k6lg krb5kdc[7720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:41:50.569633 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:50.556409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47626 (local address 127.5.225.190:43537)
0620 18:41:50.556585 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:50.556589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:50.556881 (+ 292us) server_negotiation.cc:407] Connection header received
0620 18:41:50.557071 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:50.557076 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:50.557142 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:50.557241 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:50.558593 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.559322 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.560703 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.561110 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.561208 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:50.564989 (+ 3781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:50.564995 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:50.564998 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:50.565031 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:50.567448 (+ 2417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:50.568019 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:50.568023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:50.568025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:50.568137 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:50.568446 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:50.568449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:50.568451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:50.568728 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:50.568895 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:50.569167 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 18:41:50.569324 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:41:50.573679 7765 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47626:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:41:50.580394 7765 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:41:50.603885 8268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:50.593594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37966 (local address 127.5.225.129:37425)
0620 18:41:50.595050 (+ 1456us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:50.595059 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:50.595421 (+ 362us) server_negotiation.cc:407] Connection header received
0620 18:41:50.595521 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:50.595528 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:50.595763 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:50.595955 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:50.596816 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.598249 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.601532 (+ 3283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.602518 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.602682 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:50.602900 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:50.603389 (+ 489us) server_negotiation.cc:299] Negotiation successful
0620 18:41:50.603555 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1282,"thread_start_us":1142,"threads_started":1}
I20260620 18:41:50.604045 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:50.591969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.130:42107 (local address 127.0.0.1:53390)
0620 18:41:50.594290 (+ 2321us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:50.594325 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:50.594492 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:50.595995 (+ 1503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:50.596001 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:50.596026 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:50.596530 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:50.596544 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.598272 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.598276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:50.601246 (+ 2970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:50.601260 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.603168 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.603174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:50.603392 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:50.603420 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 18:41:50.603505 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 18:41:50.603628 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2019,"mutex_wait_us":159,"spinlock_wait_cycles":1280,"thread_start_us":2016,"threads_started":1}
I20260620 18:41:50.604089 8265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:50.592325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53390 (local address 127.5.225.130:42107)
0620 18:41:50.595458 (+ 3133us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:50.595467 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:50.595488 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:41:50.595565 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:50.595571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:50.595763 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:50.596743 (+ 980us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:50.596923 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.598085 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.601401 (+ 3316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.602771 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.603372 (+ 601us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:50.603573 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:50.603648 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 18:41:50.603818 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2980,"thread_start_us":2849,"threads_started":1}
I20260620 18:41:50.604545 8267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:50.593050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:37425 (local address 127.0.0.1:37966)
0620 18:41:50.595161 (+ 2111us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:50.595185 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:50.595300 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:50.596161 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:50.596170 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:50.596203 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:50.596628 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:50.596640 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.598382 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.598387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:50.601393 (+ 3006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:50.601403 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.602720 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.602725 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:50.602910 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:50.602965 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 18:41:50.603132 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 18:41:50.604125 (+ 993us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1984,"mutex_wait_us":794,"spinlock_wait_cycles":6400,"thread_start_us":1866,"threads_started":1}
I20260620 18:41:50.605120 8264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:50.591970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.131:46387 (local address 127.0.0.1:42656)
0620 18:41:50.594373 (+ 2403us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:50.594393 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:50.594590 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:50.595471 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:50.595478 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:50.595507 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:50.595936 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:50.595948 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.598212 (+ 2264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.598219 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:50.601004 (+ 2785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:50.601022 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.603257 (+ 2235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.603263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:50.604507 (+ 1244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:50.604544 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 18:41:50.604648 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 18:41:50.604774 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2259,"mutex_wait_us":827,"thread_start_us":2120,"threads_started":1}
I20260620 18:41:50.605486 8263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:50.591996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42656 (local address 127.5.225.131:46387)
0620 18:41:50.593823 (+ 1827us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:50.593835 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:50.594482 (+ 647us) server_negotiation.cc:407] Connection header received
0620 18:41:50.594788 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:50.594797 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:50.595032 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:50.595246 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:50.596448 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.598037 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.601237 (+ 3200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.602761 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.602898 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:50.603126 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:50.604876 (+ 1750us) server_negotiation.cc:299] Negotiation successful
0620 18:41:50.605031 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1631,"thread_start_us":1460,"threads_started":1}
I20260620 18:41:50.608106 7902 tablet_service.cc:1467] Processing CreateTablet for tablet fbac0d3ff73545e3a349f7ca26a88b6f (DEFAULT_TABLE table=test-table [id=6179aeb4ac4f4a6e90d2c764dbe0daa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:41:50.608121 8041 tablet_service.cc:1467] Processing CreateTablet for tablet fbac0d3ff73545e3a349f7ca26a88b6f (DEFAULT_TABLE table=test-table [id=6179aeb4ac4f4a6e90d2c764dbe0daa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:41:50.608287 8180 tablet_service.cc:1467] Processing CreateTablet for tablet fbac0d3ff73545e3a349f7ca26a88b6f (DEFAULT_TABLE table=test-table [id=6179aeb4ac4f4a6e90d2c764dbe0daa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:41:50.609649 8041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbac0d3ff73545e3a349f7ca26a88b6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:50.609656 8180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbac0d3ff73545e3a349f7ca26a88b6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:50.609982 7902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbac0d3ff73545e3a349f7ca26a88b6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:50.621389 8269 tablet_bootstrap.cc:492] T fbac0d3ff73545e3a349f7ca26a88b6f P f6141944f9d7444dae7291f7e93a0db5: Bootstrap starting.
I20260620 18:41:50.623983 8270 tablet_bootstrap.cc:492] T fbac0d3ff73545e3a349f7ca26a88b6f P aee666d8594f44b09b3d8f73784f3979: Bootstrap starting.
I20260620 18:41:50.625247 8271 tablet_bootstrap.cc:492] T fbac0d3ff73545e3a349f7ca26a88b6f P b354f0c3d7444c3281180585aeed4c5b: Bootstrap starting.
I20260620 18:41:50.626075 8269 tablet_bootstrap.cc:654] T fbac0d3ff73545e3a349f7ca26a88b6f P f6141944f9d7444dae7291f7e93a0db5: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:50.627293 8270 tablet_bootstrap.cc:654] T fbac0d3ff73545e3a349f7ca26a88b6f P aee666d8594f44b09b3d8f73784f3979: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:50.627375 8269 log.cc:824] T fbac0d3ff73545e3a349f7ca26a88b6f P f6141944f9d7444dae7291f7e93a0db5: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:50.628405 8271 tablet_bootstrap.cc:654] T fbac0d3ff73545e3a349f7ca26a88b6f P b354f0c3d7444c3281180585aeed4c5b: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:50.628518 8270 log.cc:824] T fbac0d3ff73545e3a349f7ca26a88b6f P aee666d8594f44b09b3d8f73784f3979: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:50.629716 8271 log.cc:824] T fbac0d3ff73545e3a349f7ca26a88b6f P b354f0c3d7444c3281180585aeed4c5b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:50.630913 8269 tablet_bootstrap.cc:492] T fbac0d3ff73545e3a349f7ca26a88b6f P f6141944f9d7444dae7291f7e93a0db5: No bootstrap required, opened a new log
I20260620 18:41:50.631227 8269 ts_tablet_manager.cc:1397] T fbac0d3ff73545e3a349f7ca26a88b6f P f6141944f9d7444dae7291f7e93a0db5: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 18:41:50.631441 8270 tablet_bootstrap.cc:492] T fbac0d3ff73545e3a349f7ca26a88b6f P aee666d8594f44b09b3d8f73784f3979: No bootstrap required, opened a new log
I20260620 18:41:50.631769 8270 ts_tablet_manager.cc:1397] T fbac0d3ff73545e3a349f7ca26a88b6f P aee666d8594f44b09b3d8f73784f3979: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 18:41:50.633106 8271 tablet_bootstrap.cc:492] T fbac0d3ff73545e3a349f7ca26a88b6f P b354f0c3d7444c3281180585aeed4c5b: No bootstrap required, opened a new log
I20260620 18:41:50.633345 8271 ts_tablet_manager.cc:1397] T fbac0d3ff73545e3a349f7ca26a88b6f P b354f0c3d7444c3281180585aeed4c5b: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 18:41:50.638926 8270 raft_consensus.cc:348] T fbac0d3ff73545e3a349f7ca26a88b6f P aee666d8594f44b09b3d8f73784f3979 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } }
I20260620 18:41:50.639285 8270 raft_consensus.cc:374] T fbac0d3ff73545e3a349f7ca26a88b6f P aee666d8594f44b09b3d8f73784f3979 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:50.639401 8270 raft_consensus.cc:729] T fbac0d3ff73545e3a349f7ca26a88b6f P aee666d8594f44b09b3d8f73784f3979 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aee666d8594f44b09b3d8f73784f3979, State: Initialized, Role: FOLLOWER
I20260620 18:41:50.639945 8270 consensus_queue.cc:260] T fbac0d3ff73545e3a349f7ca26a88b6f P aee666d8594f44b09b3d8f73784f3979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } }
I20260620 18:41:50.640961 8111 heartbeater.cc:503] Master 127.5.225.190:43537 was elected leader, sending a full tablet report...
I20260620 18:41:50.641647 8270 ts_tablet_manager.cc:1428] T fbac0d3ff73545e3a349f7ca26a88b6f P aee666d8594f44b09b3d8f73784f3979: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
I20260620 18:41:50.642174 8269 raft_consensus.cc:348] T fbac0d3ff73545e3a349f7ca26a88b6f P f6141944f9d7444dae7291f7e93a0db5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } }
I20260620 18:41:50.642642 8269 raft_consensus.cc:374] T fbac0d3ff73545e3a349f7ca26a88b6f P f6141944f9d7444dae7291f7e93a0db5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:50.642781 8269 raft_consensus.cc:729] T fbac0d3ff73545e3a349f7ca26a88b6f P f6141944f9d7444dae7291f7e93a0db5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6141944f9d7444dae7291f7e93a0db5, State: Initialized, Role: FOLLOWER
I20260620 18:41:50.643471 8269 consensus_queue.cc:260] T fbac0d3ff73545e3a349f7ca26a88b6f P f6141944f9d7444dae7291f7e93a0db5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } }
I20260620 18:41:50.643637 8271 raft_consensus.cc:348] T fbac0d3ff73545e3a349f7ca26a88b6f P b354f0c3d7444c3281180585aeed4c5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } }
I20260620 18:41:50.644042 8271 raft_consensus.cc:374] T fbac0d3ff73545e3a349f7ca26a88b6f P b354f0c3d7444c3281180585aeed4c5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:50.644177 8271 raft_consensus.cc:729] T fbac0d3ff73545e3a349f7ca26a88b6f P b354f0c3d7444c3281180585aeed4c5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b354f0c3d7444c3281180585aeed4c5b, State: Initialized, Role: FOLLOWER
I20260620 18:41:50.644863 8271 consensus_queue.cc:260] T fbac0d3ff73545e3a349f7ca26a88b6f P b354f0c3d7444c3281180585aeed4c5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } }
I20260620 18:41:50.645376 8250 heartbeater.cc:503] Master 127.5.225.190:43537 was elected leader, sending a full tablet report...
I20260620 18:41:50.646641 8271 ts_tablet_manager.cc:1428] T fbac0d3ff73545e3a349f7ca26a88b6f P b354f0c3d7444c3281180585aeed4c5b: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260620 18:41:50.647264 8269 ts_tablet_manager.cc:1428] T fbac0d3ff73545e3a349f7ca26a88b6f P f6141944f9d7444dae7291f7e93a0db5: Time spent starting tablet: real 0.016s user 0.010s sys 0.004s
I20260620 18:41:50.650014 8277 raft_consensus.cc:482] T fbac0d3ff73545e3a349f7ca26a88b6f P b354f0c3d7444c3281180585aeed4c5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:41:50.650305 8277 raft_consensus.cc:504] T fbac0d3ff73545e3a349f7ca26a88b6f P b354f0c3d7444c3281180585aeed4c5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } }
I20260620 18:41:50.651839 8277 leader_election.cc:288] T fbac0d3ff73545e3a349f7ca26a88b6f P b354f0c3d7444c3281180585aeed4c5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6141944f9d7444dae7291f7e93a0db5 (127.5.225.131:46387), aee666d8594f44b09b3d8f73784f3979 (127.5.225.130:42107)
I20260620 18:41:50.659907 8265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:50.652825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:35233 (local address 127.5.225.130:42107)
0620 18:41:50.653009 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:50.653015 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:50.653298 (+ 283us) server_negotiation.cc:407] Connection header received
0620 18:41:50.653509 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:50.653515 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:50.653581 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:50.653676 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:50.654346 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.655110 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.657459 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.658739 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.658883 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:50.658959 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:50.659516 (+ 557us) server_negotiation.cc:299] Negotiation successful
0620 18:41:50.659652 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 18:41:50.659899 8263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:50.652154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:46431 (local address 127.5.225.131:46387)
0620 18:41:50.652353 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:50.652360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:50.653031 (+ 671us) server_negotiation.cc:407] Connection header received
0620 18:41:50.653242 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:50.653247 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:50.653321 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:50.653401 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:50.653960 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.654814 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.657442 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.658721 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.658900 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:50.658960 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:50.659523 (+ 563us) server_negotiation.cc:299] Negotiation successful
0620 18:41:50.659651 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 18:41:50.660120 8279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:50.652278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.130:42107 (local address 127.5.225.129:35233)
0620 18:41:50.653209 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:50.653223 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:50.653379 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:50.653691 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:50.653695 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:50.653718 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:50.654197 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:50.654209 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.655242 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.655252 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:50.657277 (+ 2025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:50.657286 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.659021 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.659028 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:50.659260 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:50.659347 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 18:41:50.659440 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 18:41:50.659632 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"mutex_wait_us":125,"thread_start_us":595,"threads_started":1}
I20260620 18:41:50.660197 8278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:50.652187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.131:46387 (local address 127.5.225.129:46431)
0620 18:41:50.652932 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:50.652948 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:50.653100 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:50.653414 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:50.653419 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:50.653444 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:50.653809 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:50.653818 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.654934 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.654941 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:50.657244 (+ 2303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:50.657262 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.659035 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.659041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:50.659247 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:50.659344 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 18:41:50.659440 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 18:41:50.659632 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":105,"thread_start_us":419,"threads_started":1}
I20260620 18:41:50.661216 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbac0d3ff73545e3a349f7ca26a88b6f" candidate_uuid: "b354f0c3d7444c3281180585aeed4c5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aee666d8594f44b09b3d8f73784f3979" is_pre_election: true
I20260620 18:41:50.661226 8202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbac0d3ff73545e3a349f7ca26a88b6f" candidate_uuid: "b354f0c3d7444c3281180585aeed4c5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6141944f9d7444dae7291f7e93a0db5" is_pre_election: true
I20260620 18:41:50.661681 8061 raft_consensus.cc:2454] T fbac0d3ff73545e3a349f7ca26a88b6f P aee666d8594f44b09b3d8f73784f3979 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b354f0c3d7444c3281180585aeed4c5b in term 0.
I20260620 18:41:50.661737 8202 raft_consensus.cc:2454] T fbac0d3ff73545e3a349f7ca26a88b6f P f6141944f9d7444dae7291f7e93a0db5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b354f0c3d7444c3281180585aeed4c5b in term 0.
I20260620 18:41:50.662346 7855 leader_election.cc:302] T fbac0d3ff73545e3a349f7ca26a88b6f P b354f0c3d7444c3281180585aeed4c5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aee666d8594f44b09b3d8f73784f3979, b354f0c3d7444c3281180585aeed4c5b; no voters:
I20260620 18:41:50.662740 8277 raft_consensus.cc:2781] T fbac0d3ff73545e3a349f7ca26a88b6f P b354f0c3d7444c3281180585aeed4c5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:41:50.662875 8277 raft_consensus.cc:482] T fbac0d3ff73545e3a349f7ca26a88b6f P b354f0c3d7444c3281180585aeed4c5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:41:50.662977 8277 raft_consensus.cc:3037] T fbac0d3ff73545e3a349f7ca26a88b6f P b354f0c3d7444c3281180585aeed4c5b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:50.665124 8277 raft_consensus.cc:504] T fbac0d3ff73545e3a349f7ca26a88b6f P b354f0c3d7444c3281180585aeed4c5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } }
I20260620 18:41:50.665764 8277 leader_election.cc:288] T fbac0d3ff73545e3a349f7ca26a88b6f P b354f0c3d7444c3281180585aeed4c5b [CANDIDATE]: Term 1 election: Requested vote from peers f6141944f9d7444dae7291f7e93a0db5 (127.5.225.131:46387), aee666d8594f44b09b3d8f73784f3979 (127.5.225.130:42107)
I20260620 18:41:50.666095 8202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbac0d3ff73545e3a349f7ca26a88b6f" candidate_uuid: "b354f0c3d7444c3281180585aeed4c5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6141944f9d7444dae7291f7e93a0db5"
I20260620 18:41:50.666240 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbac0d3ff73545e3a349f7ca26a88b6f" candidate_uuid: "b354f0c3d7444c3281180585aeed4c5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aee666d8594f44b09b3d8f73784f3979"
I20260620 18:41:50.666312 8202 raft_consensus.cc:3037] T fbac0d3ff73545e3a349f7ca26a88b6f P f6141944f9d7444dae7291f7e93a0db5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:50.666437 8061 raft_consensus.cc:3037] T fbac0d3ff73545e3a349f7ca26a88b6f P aee666d8594f44b09b3d8f73784f3979 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:50.669564 8202 raft_consensus.cc:2454] T fbac0d3ff73545e3a349f7ca26a88b6f P f6141944f9d7444dae7291f7e93a0db5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b354f0c3d7444c3281180585aeed4c5b in term 1.
I20260620 18:41:50.669564 8061 raft_consensus.cc:2454] T fbac0d3ff73545e3a349f7ca26a88b6f P aee666d8594f44b09b3d8f73784f3979 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b354f0c3d7444c3281180585aeed4c5b in term 1.
I20260620 18:41:50.669963 7852 leader_election.cc:302] T fbac0d3ff73545e3a349f7ca26a88b6f P b354f0c3d7444c3281180585aeed4c5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b354f0c3d7444c3281180585aeed4c5b, f6141944f9d7444dae7291f7e93a0db5; no voters:
I20260620 18:41:50.670285 8277 raft_consensus.cc:2781] T fbac0d3ff73545e3a349f7ca26a88b6f P b354f0c3d7444c3281180585aeed4c5b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:41:50.670728 8277 raft_consensus.cc:686] T fbac0d3ff73545e3a349f7ca26a88b6f P b354f0c3d7444c3281180585aeed4c5b [term 1 LEADER]: Becoming Leader. State: Replica: b354f0c3d7444c3281180585aeed4c5b, State: Running, Role: LEADER
I20260620 18:41:50.671231 8277 consensus_queue.cc:237] T fbac0d3ff73545e3a349f7ca26a88b6f P b354f0c3d7444c3281180585aeed4c5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } }
I20260620 18:41:50.675719 7765 catalog_manager.cc:5495] T fbac0d3ff73545e3a349f7ca26a88b6f P b354f0c3d7444c3281180585aeed4c5b reported cstate change: term changed from 0 to 1, leader changed from <none> to b354f0c3d7444c3281180585aeed4c5b (127.5.225.129). New cstate: current_term: 1 leader_uuid: "b354f0c3d7444c3281180585aeed4c5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } health_report { overall_health: HEALTHY } } }
Jun 20 18:41:50 dist-test-slave-k6lg krb5kdc[7720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980908, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:41:50.730238 8287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:50.716714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:43537 (local address 127.0.0.1:47632)
0620 18:41:50.717301 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:50.717316 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:50.717486 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:50.717827 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:50.717830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:50.718609 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:50.718874 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:50.718884 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.719900 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.719909 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:50.720879 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:50.720890 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.721640 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.721647 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:50.721819 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:50.722499 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:50.722518 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:50.724835 (+ 2317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:50.727473 (+ 2638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:50.727485 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:50.727493 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:50.727922 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:50.728522 (+ 600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:50.728526 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:50.728528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:50.728655 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:50.729255 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:50.729261 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:50.729454 (+ 193us) client_negotiation.cc:769] Sending connection context
0620 18:41:50.729676 (+ 222us) client_negotiation.cc:240] Negotiation successful
0620 18:41:50.729869 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":327,"threads_started":1}
I20260620 18:41:50.730316 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:50.716689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47632 (local address 127.5.225.190:43537)
0620 18:41:50.716873 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:50.716877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:50.717407 (+ 530us) server_negotiation.cc:407] Connection header received
0620 18:41:50.717616 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:50.717622 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:50.717705 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:50.717808 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:50.719017 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.719770 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.721017 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.721453 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.721600 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:50.725001 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:50.725006 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:50.725011 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:50.725039 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:50.727338 (+ 2299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:50.728109 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:50.728115 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:50.728119 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:50.728263 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:50.728751 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:50.728755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:50.728757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:50.729116 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:50.729268 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:50.729765 (+ 497us) server_negotiation.cc:299] Negotiation successful
0620 18:41:50.729986 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":60}
I20260620 18:41:50.735221 7765 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:47632:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 18:41:50.736287 7765 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:41:50.742502 7902 tablet_service.cc:1467] Processing CreateTablet for tablet 2bf58a3ddf2b4bb0a9c125f842d0ba69 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=52b076e1b002455eb1f74db10476ef67]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:41:50.742620 8041 tablet_service.cc:1467] Processing CreateTablet for tablet 2bf58a3ddf2b4bb0a9c125f842d0ba69 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=52b076e1b002455eb1f74db10476ef67]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:41:50.742720 8180 tablet_service.cc:1467] Processing CreateTablet for tablet 2bf58a3ddf2b4bb0a9c125f842d0ba69 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=52b076e1b002455eb1f74db10476ef67]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:41:50.743351 7902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bf58a3ddf2b4bb0a9c125f842d0ba69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:50.743522 8180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bf58a3ddf2b4bb0a9c125f842d0ba69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:50.743582 8041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bf58a3ddf2b4bb0a9c125f842d0ba69. 1 dirs total, 0 dirs full, 0 dirs failed
W20260620 18:41:50.746920 8112 tablet.cc:2369] T fbac0d3ff73545e3a349f7ca26a88b6f P aee666d8594f44b09b3d8f73784f3979: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:41:50.748749 8271 tablet_bootstrap.cc:492] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P b354f0c3d7444c3281180585aeed4c5b: Bootstrap starting.
I20260620 18:41:50.750376 8271 tablet_bootstrap.cc:654] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P b354f0c3d7444c3281180585aeed4c5b: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:50.750656 8269 tablet_bootstrap.cc:492] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P f6141944f9d7444dae7291f7e93a0db5: Bootstrap starting.
I20260620 18:41:50.751197 8270 tablet_bootstrap.cc:492] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P aee666d8594f44b09b3d8f73784f3979: Bootstrap starting.
I20260620 18:41:50.752218 8269 tablet_bootstrap.cc:654] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P f6141944f9d7444dae7291f7e93a0db5: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:50.752749 8270 tablet_bootstrap.cc:654] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P aee666d8594f44b09b3d8f73784f3979: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:50.752979 8271 tablet_bootstrap.cc:492] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P b354f0c3d7444c3281180585aeed4c5b: No bootstrap required, opened a new log
I20260620 18:41:50.753118 8271 ts_tablet_manager.cc:1397] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P b354f0c3d7444c3281180585aeed4c5b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260620 18:41:50.753957 8271 raft_consensus.cc:348] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P b354f0c3d7444c3281180585aeed4c5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } }
I20260620 18:41:50.754262 8271 raft_consensus.cc:374] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P b354f0c3d7444c3281180585aeed4c5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:50.754374 8271 raft_consensus.cc:729] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P b354f0c3d7444c3281180585aeed4c5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b354f0c3d7444c3281180585aeed4c5b, State: Initialized, Role: FOLLOWER
I20260620 18:41:50.754733 8269 tablet_bootstrap.cc:492] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P f6141944f9d7444dae7291f7e93a0db5: No bootstrap required, opened a new log
I20260620 18:41:50.754619 8271 consensus_queue.cc:260] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P b354f0c3d7444c3281180585aeed4c5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } }
I20260620 18:41:50.754868 8269 ts_tablet_manager.cc:1397] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P f6141944f9d7444dae7291f7e93a0db5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 18:41:50.755254 8271 ts_tablet_manager.cc:1428] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P b354f0c3d7444c3281180585aeed4c5b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:41:50.755134 8277 tablet_replica.cc:439] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P b354f0c3d7444c3281180585aeed4c5b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } } }
I20260620 18:41:50.755551 8277 tablet_replica.cc:442] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P b354f0c3d7444c3281180585aeed4c5b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:41:50.755647 8269 raft_consensus.cc:348] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P f6141944f9d7444dae7291f7e93a0db5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } }
I20260620 18:41:50.755882 8269 raft_consensus.cc:374] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P f6141944f9d7444dae7291f7e93a0db5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:50.755971 8269 raft_consensus.cc:729] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P f6141944f9d7444dae7291f7e93a0db5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6141944f9d7444dae7291f7e93a0db5, State: Initialized, Role: FOLLOWER
I20260620 18:41:50.756843 8270 tablet_bootstrap.cc:492] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P aee666d8594f44b09b3d8f73784f3979: No bootstrap required, opened a new log
I20260620 18:41:50.756857 8269 consensus_queue.cc:260] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P f6141944f9d7444dae7291f7e93a0db5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } }
I20260620 18:41:50.757030 8270 ts_tablet_manager.cc:1397] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P aee666d8594f44b09b3d8f73784f3979: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 18:41:50.757355 8276 tablet_replica.cc:439] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P f6141944f9d7444dae7291f7e93a0db5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } } }
I20260620 18:41:50.757738 8276 tablet_replica.cc:442] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P f6141944f9d7444dae7291f7e93a0db5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:41:50.757789 8270 raft_consensus.cc:348] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P aee666d8594f44b09b3d8f73784f3979 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } }
I20260620 18:41:50.758077 8270 raft_consensus.cc:374] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P aee666d8594f44b09b3d8f73784f3979 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:50.758208 8270 raft_consensus.cc:729] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P aee666d8594f44b09b3d8f73784f3979 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aee666d8594f44b09b3d8f73784f3979, State: Initialized, Role: FOLLOWER
I20260620 18:41:50.758363 8269 ts_tablet_manager.cc:1428] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P f6141944f9d7444dae7291f7e93a0db5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 18:41:50.758435 8270 consensus_queue.cc:260] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P aee666d8594f44b09b3d8f73784f3979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } }
I20260620 18:41:50.759096 8270 ts_tablet_manager.cc:1428] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P aee666d8594f44b09b3d8f73784f3979: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 18:41:50.758925 8275 tablet_replica.cc:439] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P aee666d8594f44b09b3d8f73784f3979: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } } }
I20260620 18:41:50.759290 8275 tablet_replica.cc:442] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P aee666d8594f44b09b3d8f73784f3979: This TxnStatusTablet replica's current role is: FOLLOWER
W20260620 18:41:50.761785 7973 tablet.cc:2369] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P b354f0c3d7444c3281180585aeed4c5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:41:50.762691 8251 tablet.cc:2369] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P f6141944f9d7444dae7291f7e93a0db5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:41:50.763688 8275 raft_consensus.cc:482] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P aee666d8594f44b09b3d8f73784f3979 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:41:50.763834 8275 raft_consensus.cc:504] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P aee666d8594f44b09b3d8f73784f3979 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } }
I20260620 18:41:50.765290 8275 leader_election.cc:288] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P aee666d8594f44b09b3d8f73784f3979 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b354f0c3d7444c3281180585aeed4c5b (127.5.225.129:37425), f6141944f9d7444dae7291f7e93a0db5 (127.5.225.131:46387)
I20260620 18:41:50.773149 8291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:50.765415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:37425 (local address 127.5.225.130:36131)
0620 18:41:50.766042 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:50.766072 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:50.766217 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:50.766759 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:50.766763 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:50.766792 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:50.767097 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:50.767106 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.768868 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.768873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:50.771019 (+ 2146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:50.771036 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.772602 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.772607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:50.772715 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:50.772739 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 18:41:50.772811 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 18:41:50.772892 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":393,"threads_started":1}
I20260620 18:41:50.773218 8268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:50.766133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:36131 (local address 127.5.225.129:37425)
0620 18:41:50.766371 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:50.766380 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:50.766406 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:41:50.766482 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:50.766486 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:50.766581 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:50.766708 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:50.767269 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.768684 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.771180 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.772385 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.772560 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:50.772633 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:50.772856 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 18:41:50.772990 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:41:50.774246 7922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bf58a3ddf2b4bb0a9c125f842d0ba69" candidate_uuid: "aee666d8594f44b09b3d8f73784f3979" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b354f0c3d7444c3281180585aeed4c5b" is_pre_election: true
I20260620 18:41:50.774674 7922 raft_consensus.cc:2454] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P b354f0c3d7444c3281180585aeed4c5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aee666d8594f44b09b3d8f73784f3979 in term 0.
I20260620 18:41:50.775281 7994 leader_election.cc:302] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P aee666d8594f44b09b3d8f73784f3979 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aee666d8594f44b09b3d8f73784f3979, b354f0c3d7444c3281180585aeed4c5b; no voters:
I20260620 18:41:50.775673 8275 raft_consensus.cc:2781] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P aee666d8594f44b09b3d8f73784f3979 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:41:50.775811 8275 raft_consensus.cc:482] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P aee666d8594f44b09b3d8f73784f3979 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:41:50.775866 8263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:50.765759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:40981 (local address 127.5.225.131:46387)
0620 18:41:50.765976 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:50.765982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:50.769247 (+ 3265us) server_negotiation.cc:407] Connection header received
0620 18:41:50.769538 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:50.769543 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:50.769613 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:50.769700 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:50.770273 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.771302 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.773641 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.774912 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.775083 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:50.775149 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:50.775487 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 18:41:50.775619 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:41:50.775911 8275 raft_consensus.cc:3037] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P aee666d8594f44b09b3d8f73784f3979 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:50.775966 8292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:50.765717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.131:46387 (local address 127.5.225.130:40981)
0620 18:41:50.769134 (+ 3417us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:50.769152 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:50.769292 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:50.769724 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:50.769727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:50.769748 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:50.770100 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:50.770108 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.771432 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.771438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:50.773482 (+ 2044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:50.773499 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.775145 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.775151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:50.775314 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:50.775347 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 18:41:50.775461 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 18:41:50.775591 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3284,"thread_start_us":3068,"threads_started":1}
I20260620 18:41:50.776546 8202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bf58a3ddf2b4bb0a9c125f842d0ba69" candidate_uuid: "aee666d8594f44b09b3d8f73784f3979" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6141944f9d7444dae7291f7e93a0db5" is_pre_election: true
I20260620 18:41:50.776772 8202 raft_consensus.cc:2454] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P f6141944f9d7444dae7291f7e93a0db5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aee666d8594f44b09b3d8f73784f3979 in term 0.
I20260620 18:41:50.778579 8275 raft_consensus.cc:504] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P aee666d8594f44b09b3d8f73784f3979 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } }
I20260620 18:41:50.779134 8275 leader_election.cc:288] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P aee666d8594f44b09b3d8f73784f3979 [CANDIDATE]: Term 1 election: Requested vote from peers b354f0c3d7444c3281180585aeed4c5b (127.5.225.129:37425), f6141944f9d7444dae7291f7e93a0db5 (127.5.225.131:46387)
I20260620 18:41:50.779578 7922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bf58a3ddf2b4bb0a9c125f842d0ba69" candidate_uuid: "aee666d8594f44b09b3d8f73784f3979" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b354f0c3d7444c3281180585aeed4c5b"
I20260620 18:41:50.779609 8202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bf58a3ddf2b4bb0a9c125f842d0ba69" candidate_uuid: "aee666d8594f44b09b3d8f73784f3979" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6141944f9d7444dae7291f7e93a0db5"
I20260620 18:41:50.779791 7922 raft_consensus.cc:3037] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P b354f0c3d7444c3281180585aeed4c5b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:50.779801 8202 raft_consensus.cc:3037] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P f6141944f9d7444dae7291f7e93a0db5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:50.782759 8202 raft_consensus.cc:2454] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P f6141944f9d7444dae7291f7e93a0db5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aee666d8594f44b09b3d8f73784f3979 in term 1.
I20260620 18:41:50.782824 7922 raft_consensus.cc:2454] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P b354f0c3d7444c3281180585aeed4c5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aee666d8594f44b09b3d8f73784f3979 in term 1.
I20260620 18:41:50.783150 7992 leader_election.cc:302] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P aee666d8594f44b09b3d8f73784f3979 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aee666d8594f44b09b3d8f73784f3979, f6141944f9d7444dae7291f7e93a0db5; no voters:
I20260620 18:41:50.783438 8275 raft_consensus.cc:2781] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P aee666d8594f44b09b3d8f73784f3979 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:41:50.783814 8275 raft_consensus.cc:686] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P aee666d8594f44b09b3d8f73784f3979 [term 1 LEADER]: Becoming Leader. State: Replica: aee666d8594f44b09b3d8f73784f3979, State: Running, Role: LEADER
I20260620 18:41:50.784255 8275 consensus_queue.cc:237] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P aee666d8594f44b09b3d8f73784f3979 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } }
I20260620 18:41:50.787513 8293 tablet_replica.cc:439] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P aee666d8594f44b09b3d8f73784f3979: TxnStatusTablet state changed. Reason: New leader aee666d8594f44b09b3d8f73784f3979. Latest consensus state: current_term: 1 leader_uuid: "aee666d8594f44b09b3d8f73784f3979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } } }
I20260620 18:41:50.787837 8293 tablet_replica.cc:442] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P aee666d8594f44b09b3d8f73784f3979: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:41:50.788519 8295 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 18:41:50.788810 8295 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 18:41:50.788879 7764 catalog_manager.cc:5495] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P aee666d8594f44b09b3d8f73784f3979 reported cstate change: term changed from 0 to 1, leader changed from <none> to aee666d8594f44b09b3d8f73784f3979 (127.5.225.130). New cstate: current_term: 1 leader_uuid: "aee666d8594f44b09b3d8f73784f3979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } health_report { overall_health: UNKNOWN } } }
I20260620 18:41:50.809013 8265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:50.803239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53402 (local address 127.5.225.130:42107)
0620 18:41:50.803397 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:50.803401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:50.803670 (+ 269us) server_negotiation.cc:407] Connection header received
0620 18:41:50.803984 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:50.803989 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:50.804077 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:50.804230 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:50.804652 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.805496 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.806839 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.807252 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.807407 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:50.807855 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:50.808322 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:50.808488 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:50.808589 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 18:41:50.808667 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 18:41:50.809015 8287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:50.803279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.130:42107 (local address 127.0.0.1:53402)
0620 18:41:50.803572 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:50.803588 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:50.803753 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:50.804250 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:50.804255 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:50.804267 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:41:50.804508 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:50.804518 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.805627 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.805634 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:50.806692 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:50.806706 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.807422 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.807428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:50.807608 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:50.807667 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:41:50.808447 (+ 780us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:41:50.808467 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 18:41:50.808556 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 18:41:50.808697 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20260620 18:41:50.816774 7922 raft_consensus.cc:1261] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P b354f0c3d7444c3281180585aeed4c5b [term 1 FOLLOWER]: Refusing update from remote peer aee666d8594f44b09b3d8f73784f3979: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:41:50.816972 8202 raft_consensus.cc:1261] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P f6141944f9d7444dae7291f7e93a0db5 [term 1 FOLLOWER]: Refusing update from remote peer aee666d8594f44b09b3d8f73784f3979: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:41:50.817730 8293 consensus_queue.cc:1035] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P aee666d8594f44b09b3d8f73784f3979 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:41:50.818073 8275 consensus_queue.cc:1035] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P aee666d8594f44b09b3d8f73784f3979 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:41:50.826501 8276 tablet_replica.cc:439] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P f6141944f9d7444dae7291f7e93a0db5: TxnStatusTablet state changed. Reason: New leader aee666d8594f44b09b3d8f73784f3979. Latest consensus state: current_term: 1 leader_uuid: "aee666d8594f44b09b3d8f73784f3979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } } }
I20260620 18:41:50.826798 8276 tablet_replica.cc:442] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P f6141944f9d7444dae7291f7e93a0db5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:41:50.828855 8298 mvcc.cc:205] Tried to move back new op lower bound from 7298993810696929280 to 7298993810575966208. Current Snapshot: MvccSnapshot[applied={T|T < 7298993810696929280}]
I20260620 18:41:50.835774 8275 tablet_replica.cc:439] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P aee666d8594f44b09b3d8f73784f3979: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aee666d8594f44b09b3d8f73784f3979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } } }
I20260620 18:41:50.836069 8275 tablet_replica.cc:442] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P aee666d8594f44b09b3d8f73784f3979: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:41:50.839705 8276 tablet_replica.cc:439] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P f6141944f9d7444dae7291f7e93a0db5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aee666d8594f44b09b3d8f73784f3979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } } }
I20260620 18:41:50.839939 8276 tablet_replica.cc:442] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P f6141944f9d7444dae7291f7e93a0db5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:41:50.840827 8277 tablet_replica.cc:439] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P b354f0c3d7444c3281180585aeed4c5b: TxnStatusTablet state changed. Reason: New leader aee666d8594f44b09b3d8f73784f3979. Latest consensus state: current_term: 1 leader_uuid: "aee666d8594f44b09b3d8f73784f3979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } } }
I20260620 18:41:50.841136 8277 tablet_replica.cc:442] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P b354f0c3d7444c3281180585aeed4c5b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:41:50.843689 8275 tablet_replica.cc:439] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P aee666d8594f44b09b3d8f73784f3979: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aee666d8594f44b09b3d8f73784f3979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } } }
I20260620 18:41:50.843951 8275 tablet_replica.cc:442] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P aee666d8594f44b09b3d8f73784f3979: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:41:50.846342 8277 tablet_replica.cc:439] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P b354f0c3d7444c3281180585aeed4c5b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aee666d8594f44b09b3d8f73784f3979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } } }
I20260620 18:41:50.846629 8277 tablet_replica.cc:442] T 2bf58a3ddf2b4bb0a9c125f842d0ba69 P b354f0c3d7444c3281180585aeed4c5b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:41:50.856534 8268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:50.850193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37974 (local address 127.5.225.129:37425)
0620 18:41:50.850369 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:50.850373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:50.850547 (+ 174us) server_negotiation.cc:407] Connection header received
0620 18:41:50.850761 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:50.850765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:50.850851 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:50.850983 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:50.851399 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.852176 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.854156 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.854736 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.854841 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:50.855316 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:50.855879 (+ 563us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:50.855986 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:50.856127 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 18:41:50.856193 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 18:41:50.874045 8265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:50.867908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:56407 (local address 127.5.225.130:42107)
0620 18:41:50.868121 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:50.868128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:50.868672 (+ 544us) server_negotiation.cc:407] Connection header received
0620 18:41:50.868827 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:50.868833 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:50.868918 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:50.869023 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:50.869490 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.870677 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.872070 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.872758 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.872886 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:50.873289 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:50.873421 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:50.873510 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:50.873680 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 18:41:50.873764 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:41:50.874068 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:50.867876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.130:42107 (local address 127.5.225.129:56407)
0620 18:41:50.868577 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:50.868593 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:50.868705 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:50.869062 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:50.869065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:50.869076 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:41:50.869340 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:50.869346 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.870815 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.870823 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:50.871932 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:50.871944 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.872934 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.872938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:50.873053 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:50.873103 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:41:50.873530 (+ 427us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:41:50.873538 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 18:41:50.873654 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 18:41:50.873781 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":368,"threads_started":1}
I20260620 18:41:50.884431 8201 raft_consensus.cc:1261] T fbac0d3ff73545e3a349f7ca26a88b6f P f6141944f9d7444dae7291f7e93a0db5 [term 1 FOLLOWER]: Refusing update from remote peer b354f0c3d7444c3281180585aeed4c5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:41:50.884749 8061 raft_consensus.cc:1261] T fbac0d3ff73545e3a349f7ca26a88b6f P aee666d8594f44b09b3d8f73784f3979 [term 1 FOLLOWER]: Refusing update from remote peer b354f0c3d7444c3281180585aeed4c5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:41:50.885072 8280 consensus_queue.cc:1035] T fbac0d3ff73545e3a349f7ca26a88b6f P b354f0c3d7444c3281180585aeed4c5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:41:50.885445 8277 consensus_queue.cc:1035] T fbac0d3ff73545e3a349f7ca26a88b6f P b354f0c3d7444c3281180585aeed4c5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:41:50.890275 8300 mvcc.cc:205] Tried to move back new op lower bound from 7298993810976337920 to 7298993810113339392. Current Snapshot: MvccSnapshot[applied={T|T < 7298993810976337920}]
I20260620 18:41:50.915772 8299 mvcc.cc:205] Tried to move back new op lower bound from 7298993810976337920 to 7298993810113339392. Current Snapshot: MvccSnapshot[applied={T|T < 7298993810976337920}]
I20260620 18:41:50.935685 8268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:50.927863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:43391 (local address 127.5.225.129:37425)
0620 18:41:50.928048 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:50.928054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:50.928653 (+ 599us) server_negotiation.cc:407] Connection header received
0620 18:41:50.929058 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:50.929064 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:50.929148 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:50.929210 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:50.930056 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.931223 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.933388 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.934249 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.934375 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:50.934918 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:50.935048 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:50.935108 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:50.935348 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 18:41:50.935422 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 18:41:50.936200 8315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:50.927806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:37425 (local address 127.5.225.130:43391)
0620 18:41:50.928543 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:50.928572 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:50.928855 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:50.929413 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:50.929419 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:50.929432 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:41:50.929845 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:50.929858 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.931448 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.931454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:50.933227 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:50.933241 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.934438 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.934445 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:50.934629 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:50.934708 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:41:50.935186 (+ 478us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:41:50.935196 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 18:41:50.935591 (+ 395us) client_negotiation.cc:240] Negotiation successful
0620 18:41:50.935735 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":346,"threads_started":1}
I20260620 18:41:50.987344 7764 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:41:50.987659 7764 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:41:50.991744 7764 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6100e6394dd4f48838ad1e248e2f501: Sending DeleteTablet for 3 replicas of tablet fbac0d3ff73545e3a349f7ca26a88b6f
I20260620 18:41:50.992794 8180 tablet_service.cc:1514] Processing DeleteTablet for tablet fbac0d3ff73545e3a349f7ca26a88b6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:41:50 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:42656
I20260620 18:41:50.992924 8041 tablet_service.cc:1514] Processing DeleteTablet for tablet fbac0d3ff73545e3a349f7ca26a88b6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:41:50 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:53390
I20260620 18:41:50.994027 7902 tablet_service.cc:1514] Processing DeleteTablet for tablet fbac0d3ff73545e3a349f7ca26a88b6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:41:50 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:37966
I20260620 18:41:51.001149 7765 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47626:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:41:51.001791 7765 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:41:51.010921 7901 tablet_service.cc:1467] Processing CreateTablet for tablet 8babbe47df744299bb8373e3c8a4185f (DEFAULT_TABLE table=test-table [id=7929eeb5dd994b188e1e90df6e9cc677]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:41:51.011492 7901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8babbe47df744299bb8373e3c8a4185f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:51.011727 8040 tablet_service.cc:1467] Processing CreateTablet for tablet 8babbe47df744299bb8373e3c8a4185f (DEFAULT_TABLE table=test-table [id=7929eeb5dd994b188e1e90df6e9cc677]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:41:51.012360 8040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8babbe47df744299bb8373e3c8a4185f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:51.014197 8179 tablet_service.cc:1467] Processing CreateTablet for tablet 8babbe47df744299bb8373e3c8a4185f (DEFAULT_TABLE table=test-table [id=7929eeb5dd994b188e1e90df6e9cc677]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:41:51.014715 8179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8babbe47df744299bb8373e3c8a4185f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:51.018491 8321 tablet_replica.cc:330] T fbac0d3ff73545e3a349f7ca26a88b6f P f6141944f9d7444dae7291f7e93a0db5: stopping tablet replica
I20260620 18:41:51.019222 8321 raft_consensus.cc:2229] T fbac0d3ff73545e3a349f7ca26a88b6f P f6141944f9d7444dae7291f7e93a0db5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:41:51.019663 8271 tablet_bootstrap.cc:492] T 8babbe47df744299bb8373e3c8a4185f P b354f0c3d7444c3281180585aeed4c5b: Bootstrap starting.
I20260620 18:41:51.020691 8321 raft_consensus.cc:2258] T fbac0d3ff73545e3a349f7ca26a88b6f P f6141944f9d7444dae7291f7e93a0db5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:41:51.021284 8271 tablet_bootstrap.cc:654] T 8babbe47df744299bb8373e3c8a4185f P b354f0c3d7444c3281180585aeed4c5b: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:51.023919 8321 ts_tablet_manager.cc:1905] T fbac0d3ff73545e3a349f7ca26a88b6f P f6141944f9d7444dae7291f7e93a0db5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:41:51.024561 8271 tablet_bootstrap.cc:492] T 8babbe47df744299bb8373e3c8a4185f P b354f0c3d7444c3281180585aeed4c5b: No bootstrap required, opened a new log
I20260620 18:41:51.024788 8271 ts_tablet_manager.cc:1397] T 8babbe47df744299bb8373e3c8a4185f P b354f0c3d7444c3281180585aeed4c5b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 18:41:51.025527 8271 raft_consensus.cc:348] T 8babbe47df744299bb8373e3c8a4185f P b354f0c3d7444c3281180585aeed4c5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } }
I20260620 18:41:51.025875 8271 raft_consensus.cc:374] T 8babbe47df744299bb8373e3c8a4185f P b354f0c3d7444c3281180585aeed4c5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:51.025918 8270 tablet_bootstrap.cc:492] T 8babbe47df744299bb8373e3c8a4185f P aee666d8594f44b09b3d8f73784f3979: Bootstrap starting.
I20260620 18:41:51.026018 8271 raft_consensus.cc:729] T 8babbe47df744299bb8373e3c8a4185f P b354f0c3d7444c3281180585aeed4c5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b354f0c3d7444c3281180585aeed4c5b, State: Initialized, Role: FOLLOWER
I20260620 18:41:51.026315 8271 consensus_queue.cc:260] T 8babbe47df744299bb8373e3c8a4185f P b354f0c3d7444c3281180585aeed4c5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } }
I20260620 18:41:51.027009 8271 ts_tablet_manager.cc:1428] T 8babbe47df744299bb8373e3c8a4185f P b354f0c3d7444c3281180585aeed4c5b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:41:51.027622 8270 tablet_bootstrap.cc:654] T 8babbe47df744299bb8373e3c8a4185f P aee666d8594f44b09b3d8f73784f3979: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:51.030776 8269 tablet_bootstrap.cc:492] T 8babbe47df744299bb8373e3c8a4185f P f6141944f9d7444dae7291f7e93a0db5: Bootstrap starting.
I20260620 18:41:51.031662 8323 tablet_replica.cc:330] T fbac0d3ff73545e3a349f7ca26a88b6f P b354f0c3d7444c3281180585aeed4c5b: stopping tablet replica
I20260620 18:41:51.032207 8269 tablet_bootstrap.cc:654] T 8babbe47df744299bb8373e3c8a4185f P f6141944f9d7444dae7291f7e93a0db5: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:51.032225 8323 raft_consensus.cc:2229] T fbac0d3ff73545e3a349f7ca26a88b6f P b354f0c3d7444c3281180585aeed4c5b [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:41:51.032866 8323 raft_consensus.cc:2258] T fbac0d3ff73545e3a349f7ca26a88b6f P b354f0c3d7444c3281180585aeed4c5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:41:51.045447 8321 ts_tablet_manager.cc:1918] T fbac0d3ff73545e3a349f7ca26a88b6f P f6141944f9d7444dae7291f7e93a0db5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:41:51.045614 8321 log.cc:1196] T fbac0d3ff73545e3a349f7ca26a88b6f P f6141944f9d7444dae7291f7e93a0db5: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-2/wal/wals/fbac0d3ff73545e3a349f7ca26a88b6f
I20260620 18:41:51.046113 8321 ts_tablet_manager.cc:1939] T fbac0d3ff73545e3a349f7ca26a88b6f P f6141944f9d7444dae7291f7e93a0db5: Deleting consensus metadata
I20260620 18:41:51.047348 8270 tablet_bootstrap.cc:492] T 8babbe47df744299bb8373e3c8a4185f P aee666d8594f44b09b3d8f73784f3979: No bootstrap required, opened a new log
I20260620 18:41:51.047534 8270 ts_tablet_manager.cc:1397] T 8babbe47df744299bb8373e3c8a4185f P aee666d8594f44b09b3d8f73784f3979: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.004s
I20260620 18:41:51.048148 8323 ts_tablet_manager.cc:1905] T fbac0d3ff73545e3a349f7ca26a88b6f P b354f0c3d7444c3281180585aeed4c5b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:41:51.048400 8269 tablet_bootstrap.cc:492] T 8babbe47df744299bb8373e3c8a4185f P f6141944f9d7444dae7291f7e93a0db5: No bootstrap required, opened a new log
I20260620 18:41:51.048576 8269 ts_tablet_manager.cc:1397] T 8babbe47df744299bb8373e3c8a4185f P f6141944f9d7444dae7291f7e93a0db5: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.015s
I20260620 18:41:51.048540 8270 raft_consensus.cc:348] T 8babbe47df744299bb8373e3c8a4185f P aee666d8594f44b09b3d8f73784f3979 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } }
I20260620 18:41:51.048784 8270 raft_consensus.cc:374] T 8babbe47df744299bb8373e3c8a4185f P aee666d8594f44b09b3d8f73784f3979 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:51.048868 8270 raft_consensus.cc:729] T 8babbe47df744299bb8373e3c8a4185f P aee666d8594f44b09b3d8f73784f3979 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aee666d8594f44b09b3d8f73784f3979, State: Initialized, Role: FOLLOWER
I20260620 18:41:51.049154 8270 consensus_queue.cc:260] T 8babbe47df744299bb8373e3c8a4185f P aee666d8594f44b09b3d8f73784f3979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } }
I20260620 18:41:51.049336 8269 raft_consensus.cc:348] T 8babbe47df744299bb8373e3c8a4185f P f6141944f9d7444dae7291f7e93a0db5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } }
I20260620 18:41:51.049623 8269 raft_consensus.cc:374] T 8babbe47df744299bb8373e3c8a4185f P f6141944f9d7444dae7291f7e93a0db5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:51.049712 8269 raft_consensus.cc:729] T 8babbe47df744299bb8373e3c8a4185f P f6141944f9d7444dae7291f7e93a0db5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6141944f9d7444dae7291f7e93a0db5, State: Initialized, Role: FOLLOWER
I20260620 18:41:51.050061 8269 consensus_queue.cc:260] T 8babbe47df744299bb8373e3c8a4185f P f6141944f9d7444dae7291f7e93a0db5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } }
I20260620 18:41:51.050637 8269 ts_tablet_manager.cc:1428] T 8babbe47df744299bb8373e3c8a4185f P f6141944f9d7444dae7291f7e93a0db5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:41:51.054297 8270 ts_tablet_manager.cc:1428] T 8babbe47df744299bb8373e3c8a4185f P aee666d8594f44b09b3d8f73784f3979: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20260620 18:41:51.057117 8322 tablet_replica.cc:330] T fbac0d3ff73545e3a349f7ca26a88b6f P aee666d8594f44b09b3d8f73784f3979: stopping tablet replica
I20260620 18:41:51.057480 8322 raft_consensus.cc:2229] T fbac0d3ff73545e3a349f7ca26a88b6f P aee666d8594f44b09b3d8f73784f3979 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:41:51.057641 7749 catalog_manager.cc:4841] TS f6141944f9d7444dae7291f7e93a0db5 (127.5.225.131:46387): tablet fbac0d3ff73545e3a349f7ca26a88b6f (table test-table [id=6179aeb4ac4f4a6e90d2c764dbe0daa3]) successfully deleted
I20260620 18:41:51.057773 8322 raft_consensus.cc:2258] T fbac0d3ff73545e3a349f7ca26a88b6f P aee666d8594f44b09b3d8f73784f3979 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:41:51.059895 8323 ts_tablet_manager.cc:1918] T fbac0d3ff73545e3a349f7ca26a88b6f P b354f0c3d7444c3281180585aeed4c5b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:41:51.060155 8323 log.cc:1196] T fbac0d3ff73545e3a349f7ca26a88b6f P b354f0c3d7444c3281180585aeed4c5b: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-0/wal/wals/fbac0d3ff73545e3a349f7ca26a88b6f
I20260620 18:41:51.060740 8323 ts_tablet_manager.cc:1939] T fbac0d3ff73545e3a349f7ca26a88b6f P b354f0c3d7444c3281180585aeed4c5b: Deleting consensus metadata
I20260620 18:41:51.062455 7751 catalog_manager.cc:4841] TS b354f0c3d7444c3281180585aeed4c5b (127.5.225.129:37425): tablet fbac0d3ff73545e3a349f7ca26a88b6f (table test-table [id=6179aeb4ac4f4a6e90d2c764dbe0daa3]) successfully deleted
I20260620 18:41:51.064358 8322 ts_tablet_manager.cc:1905] T fbac0d3ff73545e3a349f7ca26a88b6f P aee666d8594f44b09b3d8f73784f3979: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:41:51.072213 8322 ts_tablet_manager.cc:1918] T fbac0d3ff73545e3a349f7ca26a88b6f P aee666d8594f44b09b3d8f73784f3979: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:41:51.072378 8322 log.cc:1196] T fbac0d3ff73545e3a349f7ca26a88b6f P aee666d8594f44b09b3d8f73784f3979: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-1/wal/wals/fbac0d3ff73545e3a349f7ca26a88b6f
I20260620 18:41:51.072923 8322 ts_tablet_manager.cc:1939] T fbac0d3ff73545e3a349f7ca26a88b6f P aee666d8594f44b09b3d8f73784f3979: Deleting consensus metadata
I20260620 18:41:51.074776 7752 catalog_manager.cc:4841] TS aee666d8594f44b09b3d8f73784f3979 (127.5.225.130:42107): tablet fbac0d3ff73545e3a349f7ca26a88b6f (table test-table [id=6179aeb4ac4f4a6e90d2c764dbe0daa3]) successfully deleted
I20260620 18:41:51.143313 8318 raft_consensus.cc:482] T 8babbe47df744299bb8373e3c8a4185f P b354f0c3d7444c3281180585aeed4c5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:41:51.143566 8318 raft_consensus.cc:504] T 8babbe47df744299bb8373e3c8a4185f P b354f0c3d7444c3281180585aeed4c5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } }
I20260620 18:41:51.144423 8318 leader_election.cc:288] T 8babbe47df744299bb8373e3c8a4185f P b354f0c3d7444c3281180585aeed4c5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aee666d8594f44b09b3d8f73784f3979 (127.5.225.130:42107), f6141944f9d7444dae7291f7e93a0db5 (127.5.225.131:46387)
I20260620 18:41:51.145238 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8babbe47df744299bb8373e3c8a4185f" candidate_uuid: "b354f0c3d7444c3281180585aeed4c5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aee666d8594f44b09b3d8f73784f3979" is_pre_election: true
I20260620 18:41:51.145310 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8babbe47df744299bb8373e3c8a4185f" candidate_uuid: "b354f0c3d7444c3281180585aeed4c5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6141944f9d7444dae7291f7e93a0db5" is_pre_election: true
I20260620 18:41:51.145628 8201 raft_consensus.cc:2454] T 8babbe47df744299bb8373e3c8a4185f P f6141944f9d7444dae7291f7e93a0db5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b354f0c3d7444c3281180585aeed4c5b in term 0.
I20260620 18:41:51.145624 8061 raft_consensus.cc:2454] T 8babbe47df744299bb8373e3c8a4185f P aee666d8594f44b09b3d8f73784f3979 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b354f0c3d7444c3281180585aeed4c5b in term 0.
I20260620 18:41:51.146119 7855 leader_election.cc:302] T 8babbe47df744299bb8373e3c8a4185f P b354f0c3d7444c3281180585aeed4c5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aee666d8594f44b09b3d8f73784f3979, b354f0c3d7444c3281180585aeed4c5b; no voters:
I20260620 18:41:51.146438 8318 raft_consensus.cc:2781] T 8babbe47df744299bb8373e3c8a4185f P b354f0c3d7444c3281180585aeed4c5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:41:51.146580 8318 raft_consensus.cc:482] T 8babbe47df744299bb8373e3c8a4185f P b354f0c3d7444c3281180585aeed4c5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:41:51.146667 8318 raft_consensus.cc:3037] T 8babbe47df744299bb8373e3c8a4185f P b354f0c3d7444c3281180585aeed4c5b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:51.150128 8318 raft_consensus.cc:504] T 8babbe47df744299bb8373e3c8a4185f P b354f0c3d7444c3281180585aeed4c5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } }
I20260620 18:41:51.153126 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8babbe47df744299bb8373e3c8a4185f" candidate_uuid: "b354f0c3d7444c3281180585aeed4c5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aee666d8594f44b09b3d8f73784f3979"
I20260620 18:41:51.153373 8061 raft_consensus.cc:3037] T 8babbe47df744299bb8373e3c8a4185f P aee666d8594f44b09b3d8f73784f3979 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:51.156595 8061 raft_consensus.cc:2454] T 8babbe47df744299bb8373e3c8a4185f P aee666d8594f44b09b3d8f73784f3979 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b354f0c3d7444c3281180585aeed4c5b in term 1.
I20260620 18:41:51.157104 7855 leader_election.cc:302] T 8babbe47df744299bb8373e3c8a4185f P b354f0c3d7444c3281180585aeed4c5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aee666d8594f44b09b3d8f73784f3979, b354f0c3d7444c3281180585aeed4c5b; no voters:
I20260620 18:41:51.157397 8280 raft_consensus.cc:2781] T 8babbe47df744299bb8373e3c8a4185f P b354f0c3d7444c3281180585aeed4c5b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:41:51.157586 8318 leader_election.cc:288] T 8babbe47df744299bb8373e3c8a4185f P b354f0c3d7444c3281180585aeed4c5b [CANDIDATE]: Term 1 election: Requested vote from peers aee666d8594f44b09b3d8f73784f3979 (127.5.225.130:42107), f6141944f9d7444dae7291f7e93a0db5 (127.5.225.131:46387)
I20260620 18:41:51.157613 8280 raft_consensus.cc:686] T 8babbe47df744299bb8373e3c8a4185f P b354f0c3d7444c3281180585aeed4c5b [term 1 LEADER]: Becoming Leader. State: Replica: b354f0c3d7444c3281180585aeed4c5b, State: Running, Role: LEADER
I20260620 18:41:51.157909 8280 consensus_queue.cc:237] T 8babbe47df744299bb8373e3c8a4185f P b354f0c3d7444c3281180585aeed4c5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } }
I20260620 18:41:51.159543 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8babbe47df744299bb8373e3c8a4185f" candidate_uuid: "b354f0c3d7444c3281180585aeed4c5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6141944f9d7444dae7291f7e93a0db5"
I20260620 18:41:51.159786 8201 raft_consensus.cc:3037] T 8babbe47df744299bb8373e3c8a4185f P f6141944f9d7444dae7291f7e93a0db5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:51.163146 8201 raft_consensus.cc:2454] T 8babbe47df744299bb8373e3c8a4185f P f6141944f9d7444dae7291f7e93a0db5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b354f0c3d7444c3281180585aeed4c5b in term 1.
I20260620 18:41:51.168342 7765 catalog_manager.cc:5495] T 8babbe47df744299bb8373e3c8a4185f P b354f0c3d7444c3281180585aeed4c5b reported cstate change: term changed from 0 to 1, leader changed from <none> to b354f0c3d7444c3281180585aeed4c5b (127.5.225.129). New cstate: current_term: 1 leader_uuid: "b354f0c3d7444c3281180585aeed4c5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } health_report { overall_health: UNKNOWN } } }
I20260620 18:41:51.203342 8201 raft_consensus.cc:1261] T 8babbe47df744299bb8373e3c8a4185f P f6141944f9d7444dae7291f7e93a0db5 [term 1 FOLLOWER]: Refusing update from remote peer b354f0c3d7444c3281180585aeed4c5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:41:51.203624 8061 raft_consensus.cc:1261] T 8babbe47df744299bb8373e3c8a4185f P aee666d8594f44b09b3d8f73784f3979 [term 1 FOLLOWER]: Refusing update from remote peer b354f0c3d7444c3281180585aeed4c5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:41:51.204008 8317 consensus_queue.cc:1035] T 8babbe47df744299bb8373e3c8a4185f P b354f0c3d7444c3281180585aeed4c5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:41:51.205039 8317 consensus_queue.cc:1035] T 8babbe47df744299bb8373e3c8a4185f P b354f0c3d7444c3281180585aeed4c5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 18:41:51.238325 8300 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 18:41:52.847688 7764 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:41:52.847899 7764 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:41:52.852221 7764 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6100e6394dd4f48838ad1e248e2f501: Sending DeleteTablet for 3 replicas of tablet 8babbe47df744299bb8373e3c8a4185f
I20260620 18:41:52.853049 7902 tablet_service.cc:1514] Processing DeleteTablet for tablet 8babbe47df744299bb8373e3c8a4185f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:41:52 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:37966
I20260620 18:41:52.853276 8041 tablet_service.cc:1514] Processing DeleteTablet for tablet 8babbe47df744299bb8373e3c8a4185f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:41:52 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:53390
I20260620 18:41:52.853590 8179 tablet_service.cc:1514] Processing DeleteTablet for tablet 8babbe47df744299bb8373e3c8a4185f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:41:52 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:42656
I20260620 18:41:52.865468 8353 tablet_replica.cc:330] T 8babbe47df744299bb8373e3c8a4185f P aee666d8594f44b09b3d8f73784f3979: stopping tablet replica
I20260620 18:41:52.865706 8353 raft_consensus.cc:2229] T 8babbe47df744299bb8373e3c8a4185f P aee666d8594f44b09b3d8f73784f3979 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:41:52.865942 8353 raft_consensus.cc:2258] T 8babbe47df744299bb8373e3c8a4185f P aee666d8594f44b09b3d8f73784f3979 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:41:52.866540 8352 tablet_replica.cc:330] T 8babbe47df744299bb8373e3c8a4185f P b354f0c3d7444c3281180585aeed4c5b: stopping tablet replica
I20260620 18:41:52.866753 8352 raft_consensus.cc:2229] T 8babbe47df744299bb8373e3c8a4185f P b354f0c3d7444c3281180585aeed4c5b [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:41:52.867091 8352 raft_consensus.cc:2258] T 8babbe47df744299bb8373e3c8a4185f P b354f0c3d7444c3281180585aeed4c5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:41:52.867663 8354 tablet_replica.cc:330] T 8babbe47df744299bb8373e3c8a4185f P f6141944f9d7444dae7291f7e93a0db5: stopping tablet replica
I20260620 18:41:52.867926 8353 ts_tablet_manager.cc:1905] T 8babbe47df744299bb8373e3c8a4185f P aee666d8594f44b09b3d8f73784f3979: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:41:52.867905 8354 raft_consensus.cc:2229] T 8babbe47df744299bb8373e3c8a4185f P f6141944f9d7444dae7291f7e93a0db5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:41:52.868165 8354 raft_consensus.cc:2258] T 8babbe47df744299bb8373e3c8a4185f P f6141944f9d7444dae7291f7e93a0db5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:41:52.869171 8354 ts_tablet_manager.cc:1905] T 8babbe47df744299bb8373e3c8a4185f P f6141944f9d7444dae7291f7e93a0db5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:41:52.870802 8352 ts_tablet_manager.cc:1905] T 8babbe47df744299bb8373e3c8a4185f P b354f0c3d7444c3281180585aeed4c5b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:41:52.873903 8353 ts_tablet_manager.cc:1918] T 8babbe47df744299bb8373e3c8a4185f P aee666d8594f44b09b3d8f73784f3979: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:41:52.874039 8353 log.cc:1196] T 8babbe47df744299bb8373e3c8a4185f P aee666d8594f44b09b3d8f73784f3979: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-1/wal/wals/8babbe47df744299bb8373e3c8a4185f
I20260620 18:41:52.874509 8353 ts_tablet_manager.cc:1939] T 8babbe47df744299bb8373e3c8a4185f P aee666d8594f44b09b3d8f73784f3979: Deleting consensus metadata
I20260620 18:41:52.875711 7752 catalog_manager.cc:4841] TS aee666d8594f44b09b3d8f73784f3979 (127.5.225.130:42107): tablet 8babbe47df744299bb8373e3c8a4185f (table test-table [id=7929eeb5dd994b188e1e90df6e9cc677]) successfully deleted
I20260620 18:41:52.877338 8354 ts_tablet_manager.cc:1918] T 8babbe47df744299bb8373e3c8a4185f P f6141944f9d7444dae7291f7e93a0db5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:41:52.877557 8354 log.cc:1196] T 8babbe47df744299bb8373e3c8a4185f P f6141944f9d7444dae7291f7e93a0db5: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-2/wal/wals/8babbe47df744299bb8373e3c8a4185f
I20260620 18:41:52.877663 8352 ts_tablet_manager.cc:1918] T 8babbe47df744299bb8373e3c8a4185f P b354f0c3d7444c3281180585aeed4c5b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:41:52.877795 8352 log.cc:1196] T 8babbe47df744299bb8373e3c8a4185f P b354f0c3d7444c3281180585aeed4c5b: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-0/wal/wals/8babbe47df744299bb8373e3c8a4185f
I20260620 18:41:52.878106 8354 ts_tablet_manager.cc:1939] T 8babbe47df744299bb8373e3c8a4185f P f6141944f9d7444dae7291f7e93a0db5: Deleting consensus metadata
I20260620 18:41:52.878294 8352 ts_tablet_manager.cc:1939] T 8babbe47df744299bb8373e3c8a4185f P b354f0c3d7444c3281180585aeed4c5b: Deleting consensus metadata
I20260620 18:41:52.879308 7749 catalog_manager.cc:4841] TS f6141944f9d7444dae7291f7e93a0db5 (127.5.225.131:46387): tablet 8babbe47df744299bb8373e3c8a4185f (table test-table [id=7929eeb5dd994b188e1e90df6e9cc677]) successfully deleted
I20260620 18:41:52.879455 7751 catalog_manager.cc:4841] TS b354f0c3d7444c3281180585aeed4c5b (127.5.225.129:37425): tablet 8babbe47df744299bb8373e3c8a4185f (table test-table [id=7929eeb5dd994b188e1e90df6e9cc677]) successfully deleted
Jun 20 18:41:52 dist-test-slave-k6lg krb5kdc[7720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980912, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 18:41:52 dist-test-slave-k6lg krb5kdc[7720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980912, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:41:52.911787 8362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:52.897820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47644 (local address 127.5.225.190:43537)
0620 18:41:52.898749 (+ 929us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:52.898754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:52.898772 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:41:52.898819 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:52.898823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:52.898883 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:52.899003 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:52.900307 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:52.901257 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:52.902613 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:52.903016 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:52.903158 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:52.906876 (+ 3718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:52.906882 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:52.906885 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:52.906918 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:52.909547 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:52.910134 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:52.910138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:52.910140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:52.910306 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:52.910710 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:52.910714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:52.910716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:52.910965 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:52.911138 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:52.911380 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 18:41:52.911531 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":783,"thread_start_us":608,"threads_started":1}
I20260620 18:41:52.915066 7764 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:41:52.915585 7764 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:41:52.921833 7902 tablet_service.cc:1467] Processing CreateTablet for tablet 0ce81cdde3ab410c99b6d898f85312e8 (DEFAULT_TABLE table=test-table [id=21938f2a37a3466ab23aa8598d3e1e17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:41:52.922326 7902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ce81cdde3ab410c99b6d898f85312e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:52.922287 8041 tablet_service.cc:1467] Processing CreateTablet for tablet 0ce81cdde3ab410c99b6d898f85312e8 (DEFAULT_TABLE table=test-table [id=21938f2a37a3466ab23aa8598d3e1e17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:41:52.922276 8179 tablet_service.cc:1467] Processing CreateTablet for tablet 0ce81cdde3ab410c99b6d898f85312e8 (DEFAULT_TABLE table=test-table [id=21938f2a37a3466ab23aa8598d3e1e17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:41:52.922693 8041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ce81cdde3ab410c99b6d898f85312e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:52.922740 8179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ce81cdde3ab410c99b6d898f85312e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:52.929785 8365 tablet_bootstrap.cc:492] T 0ce81cdde3ab410c99b6d898f85312e8 P f6141944f9d7444dae7291f7e93a0db5: Bootstrap starting.
I20260620 18:41:52.931003 8363 tablet_bootstrap.cc:492] T 0ce81cdde3ab410c99b6d898f85312e8 P aee666d8594f44b09b3d8f73784f3979: Bootstrap starting.
I20260620 18:41:52.931412 8364 tablet_bootstrap.cc:492] T 0ce81cdde3ab410c99b6d898f85312e8 P b354f0c3d7444c3281180585aeed4c5b: Bootstrap starting.
I20260620 18:41:52.932767 8364 tablet_bootstrap.cc:654] T 0ce81cdde3ab410c99b6d898f85312e8 P b354f0c3d7444c3281180585aeed4c5b: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:52.932775 8363 tablet_bootstrap.cc:654] T 0ce81cdde3ab410c99b6d898f85312e8 P aee666d8594f44b09b3d8f73784f3979: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:52.932878 8365 tablet_bootstrap.cc:654] T 0ce81cdde3ab410c99b6d898f85312e8 P f6141944f9d7444dae7291f7e93a0db5: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:52.934725 8364 tablet_bootstrap.cc:492] T 0ce81cdde3ab410c99b6d898f85312e8 P b354f0c3d7444c3281180585aeed4c5b: No bootstrap required, opened a new log
I20260620 18:41:52.934883 8364 ts_tablet_manager.cc:1397] T 0ce81cdde3ab410c99b6d898f85312e8 P b354f0c3d7444c3281180585aeed4c5b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 18:41:52.935401 8365 tablet_bootstrap.cc:492] T 0ce81cdde3ab410c99b6d898f85312e8 P f6141944f9d7444dae7291f7e93a0db5: No bootstrap required, opened a new log
I20260620 18:41:52.935585 8365 ts_tablet_manager.cc:1397] T 0ce81cdde3ab410c99b6d898f85312e8 P f6141944f9d7444dae7291f7e93a0db5: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260620 18:41:52.935701 8364 raft_consensus.cc:348] T 0ce81cdde3ab410c99b6d898f85312e8 P b354f0c3d7444c3281180585aeed4c5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } }
I20260620 18:41:52.935948 8364 raft_consensus.cc:374] T 0ce81cdde3ab410c99b6d898f85312e8 P b354f0c3d7444c3281180585aeed4c5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:52.936048 8364 raft_consensus.cc:729] T 0ce81cdde3ab410c99b6d898f85312e8 P b354f0c3d7444c3281180585aeed4c5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b354f0c3d7444c3281180585aeed4c5b, State: Initialized, Role: FOLLOWER
I20260620 18:41:52.936293 8364 consensus_queue.cc:260] T 0ce81cdde3ab410c99b6d898f85312e8 P b354f0c3d7444c3281180585aeed4c5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } }
I20260620 18:41:52.936290 8365 raft_consensus.cc:348] T 0ce81cdde3ab410c99b6d898f85312e8 P f6141944f9d7444dae7291f7e93a0db5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } }
I20260620 18:41:52.936491 8365 raft_consensus.cc:374] T 0ce81cdde3ab410c99b6d898f85312e8 P f6141944f9d7444dae7291f7e93a0db5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:52.936626 8365 raft_consensus.cc:729] T 0ce81cdde3ab410c99b6d898f85312e8 P f6141944f9d7444dae7291f7e93a0db5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6141944f9d7444dae7291f7e93a0db5, State: Initialized, Role: FOLLOWER
I20260620 18:41:52.936800 8364 ts_tablet_manager.cc:1428] T 0ce81cdde3ab410c99b6d898f85312e8 P b354f0c3d7444c3281180585aeed4c5b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:41:52.936908 8365 consensus_queue.cc:260] T 0ce81cdde3ab410c99b6d898f85312e8 P f6141944f9d7444dae7291f7e93a0db5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } }
I20260620 18:41:52.937752 8363 tablet_bootstrap.cc:492] T 0ce81cdde3ab410c99b6d898f85312e8 P aee666d8594f44b09b3d8f73784f3979: No bootstrap required, opened a new log
I20260620 18:41:52.937902 8363 ts_tablet_manager.cc:1397] T 0ce81cdde3ab410c99b6d898f85312e8 P aee666d8594f44b09b3d8f73784f3979: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20260620 18:41:52.938189 8365 ts_tablet_manager.cc:1428] T 0ce81cdde3ab410c99b6d898f85312e8 P f6141944f9d7444dae7291f7e93a0db5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 18:41:52.938730 8363 raft_consensus.cc:348] T 0ce81cdde3ab410c99b6d898f85312e8 P aee666d8594f44b09b3d8f73784f3979 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } }
I20260620 18:41:52.938963 8363 raft_consensus.cc:374] T 0ce81cdde3ab410c99b6d898f85312e8 P aee666d8594f44b09b3d8f73784f3979 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:52.939070 8363 raft_consensus.cc:729] T 0ce81cdde3ab410c99b6d898f85312e8 P aee666d8594f44b09b3d8f73784f3979 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aee666d8594f44b09b3d8f73784f3979, State: Initialized, Role: FOLLOWER
I20260620 18:41:52.939285 8363 consensus_queue.cc:260] T 0ce81cdde3ab410c99b6d898f85312e8 P aee666d8594f44b09b3d8f73784f3979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } }
I20260620 18:41:52.939852 8363 ts_tablet_manager.cc:1428] T 0ce81cdde3ab410c99b6d898f85312e8 P aee666d8594f44b09b3d8f73784f3979: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:41:53.010666 8342 raft_consensus.cc:482] T 0ce81cdde3ab410c99b6d898f85312e8 P b354f0c3d7444c3281180585aeed4c5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:41:53.010883 8342 raft_consensus.cc:504] T 0ce81cdde3ab410c99b6d898f85312e8 P b354f0c3d7444c3281180585aeed4c5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } }
I20260620 18:41:53.011577 8342 leader_election.cc:288] T 0ce81cdde3ab410c99b6d898f85312e8 P b354f0c3d7444c3281180585aeed4c5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aee666d8594f44b09b3d8f73784f3979 (127.5.225.130:42107), f6141944f9d7444dae7291f7e93a0db5 (127.5.225.131:46387)
I20260620 18:41:53.012131 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce81cdde3ab410c99b6d898f85312e8" candidate_uuid: "b354f0c3d7444c3281180585aeed4c5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6141944f9d7444dae7291f7e93a0db5" is_pre_election: true
I20260620 18:41:53.012148 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce81cdde3ab410c99b6d898f85312e8" candidate_uuid: "b354f0c3d7444c3281180585aeed4c5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aee666d8594f44b09b3d8f73784f3979" is_pre_election: true
I20260620 18:41:53.012383 8061 raft_consensus.cc:2454] T 0ce81cdde3ab410c99b6d898f85312e8 P aee666d8594f44b09b3d8f73784f3979 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b354f0c3d7444c3281180585aeed4c5b in term 0.
I20260620 18:41:53.012413 8201 raft_consensus.cc:2454] T 0ce81cdde3ab410c99b6d898f85312e8 P f6141944f9d7444dae7291f7e93a0db5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b354f0c3d7444c3281180585aeed4c5b in term 0.
I20260620 18:41:53.012871 7855 leader_election.cc:302] T 0ce81cdde3ab410c99b6d898f85312e8 P b354f0c3d7444c3281180585aeed4c5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aee666d8594f44b09b3d8f73784f3979, b354f0c3d7444c3281180585aeed4c5b; no voters:
I20260620 18:41:53.013163 8342 raft_consensus.cc:2781] T 0ce81cdde3ab410c99b6d898f85312e8 P b354f0c3d7444c3281180585aeed4c5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:41:53.013314 8342 raft_consensus.cc:482] T 0ce81cdde3ab410c99b6d898f85312e8 P b354f0c3d7444c3281180585aeed4c5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:41:53.013433 8342 raft_consensus.cc:3037] T 0ce81cdde3ab410c99b6d898f85312e8 P b354f0c3d7444c3281180585aeed4c5b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:53.015498 8342 raft_consensus.cc:504] T 0ce81cdde3ab410c99b6d898f85312e8 P b354f0c3d7444c3281180585aeed4c5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } }
I20260620 18:41:53.016079 8342 leader_election.cc:288] T 0ce81cdde3ab410c99b6d898f85312e8 P b354f0c3d7444c3281180585aeed4c5b [CANDIDATE]: Term 1 election: Requested vote from peers aee666d8594f44b09b3d8f73784f3979 (127.5.225.130:42107), f6141944f9d7444dae7291f7e93a0db5 (127.5.225.131:46387)
I20260620 18:41:53.016603 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce81cdde3ab410c99b6d898f85312e8" candidate_uuid: "b354f0c3d7444c3281180585aeed4c5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aee666d8594f44b09b3d8f73784f3979"
I20260620 18:41:53.016649 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce81cdde3ab410c99b6d898f85312e8" candidate_uuid: "b354f0c3d7444c3281180585aeed4c5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6141944f9d7444dae7291f7e93a0db5"
I20260620 18:41:53.016831 8061 raft_consensus.cc:3037] T 0ce81cdde3ab410c99b6d898f85312e8 P aee666d8594f44b09b3d8f73784f3979 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:53.016852 8201 raft_consensus.cc:3037] T 0ce81cdde3ab410c99b6d898f85312e8 P f6141944f9d7444dae7291f7e93a0db5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:53.019793 8201 raft_consensus.cc:2454] T 0ce81cdde3ab410c99b6d898f85312e8 P f6141944f9d7444dae7291f7e93a0db5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b354f0c3d7444c3281180585aeed4c5b in term 1.
I20260620 18:41:53.019806 8061 raft_consensus.cc:2454] T 0ce81cdde3ab410c99b6d898f85312e8 P aee666d8594f44b09b3d8f73784f3979 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b354f0c3d7444c3281180585aeed4c5b in term 1.
I20260620 18:41:53.020202 7852 leader_election.cc:302] T 0ce81cdde3ab410c99b6d898f85312e8 P b354f0c3d7444c3281180585aeed4c5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b354f0c3d7444c3281180585aeed4c5b, f6141944f9d7444dae7291f7e93a0db5; no voters:
I20260620 18:41:53.020524 8342 raft_consensus.cc:2781] T 0ce81cdde3ab410c99b6d898f85312e8 P b354f0c3d7444c3281180585aeed4c5b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:41:53.020689 8342 raft_consensus.cc:686] T 0ce81cdde3ab410c99b6d898f85312e8 P b354f0c3d7444c3281180585aeed4c5b [term 1 LEADER]: Becoming Leader. State: Replica: b354f0c3d7444c3281180585aeed4c5b, State: Running, Role: LEADER
I20260620 18:41:53.020956 8342 consensus_queue.cc:237] T 0ce81cdde3ab410c99b6d898f85312e8 P b354f0c3d7444c3281180585aeed4c5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } }
I20260620 18:41:53.023739 7765 catalog_manager.cc:5495] T 0ce81cdde3ab410c99b6d898f85312e8 P b354f0c3d7444c3281180585aeed4c5b reported cstate change: term changed from 0 to 1, leader changed from <none> to b354f0c3d7444c3281180585aeed4c5b (127.5.225.129). New cstate: current_term: 1 leader_uuid: "b354f0c3d7444c3281180585aeed4c5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } health_report { overall_health: UNKNOWN } } }
I20260620 18:41:53.063962 8372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:53.058799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37982 (local address 127.5.225.129:37425)
0620 18:41:53.059367 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:53.059371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:53.059389 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:41:53.059444 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:53.059448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:53.059503 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:53.059614 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:53.060016 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:53.060787 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:53.062110 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:53.062533 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:53.062669 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:53.063161 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:53.063273 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:53.063415 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:53.063595 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 18:41:53.063666 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":305,"threads_started":1}
I20260620 18:41:53.072846 8201 raft_consensus.cc:1261] T 0ce81cdde3ab410c99b6d898f85312e8 P f6141944f9d7444dae7291f7e93a0db5 [term 1 FOLLOWER]: Refusing update from remote peer b354f0c3d7444c3281180585aeed4c5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:41:53.072997 8061 raft_consensus.cc:1261] T 0ce81cdde3ab410c99b6d898f85312e8 P aee666d8594f44b09b3d8f73784f3979 [term 1 FOLLOWER]: Refusing update from remote peer b354f0c3d7444c3281180585aeed4c5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:41:53.073478 8345 consensus_queue.cc:1035] T 0ce81cdde3ab410c99b6d898f85312e8 P b354f0c3d7444c3281180585aeed4c5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:41:53.073902 8342 consensus_queue.cc:1035] T 0ce81cdde3ab410c99b6d898f85312e8 P b354f0c3d7444c3281180585aeed4c5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:41:53.117364 7763 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:41:53.117619 7763 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:41:53.120451 7763 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6100e6394dd4f48838ad1e248e2f501: Sending DeleteTablet for 3 replicas of tablet 0ce81cdde3ab410c99b6d898f85312e8
I20260620 18:41:53.121215 7902 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ce81cdde3ab410c99b6d898f85312e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:41:53 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:37966
I20260620 18:41:53.121217 8041 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ce81cdde3ab410c99b6d898f85312e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:41:53 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:53390
I20260620 18:41:53.121315 8179 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ce81cdde3ab410c99b6d898f85312e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:41:53 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:42656
I20260620 18:41:53.121479 8352 tablet_replica.cc:330] T 0ce81cdde3ab410c99b6d898f85312e8 P b354f0c3d7444c3281180585aeed4c5b: stopping tablet replica
I20260620 18:41:53.121627 8354 tablet_replica.cc:330] T 0ce81cdde3ab410c99b6d898f85312e8 P f6141944f9d7444dae7291f7e93a0db5: stopping tablet replica
I20260620 18:41:53.121703 8352 raft_consensus.cc:2229] T 0ce81cdde3ab410c99b6d898f85312e8 P b354f0c3d7444c3281180585aeed4c5b [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:41:53.121830 8354 raft_consensus.cc:2229] T 0ce81cdde3ab410c99b6d898f85312e8 P f6141944f9d7444dae7291f7e93a0db5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:41:53.122052 8354 raft_consensus.cc:2258] T 0ce81cdde3ab410c99b6d898f85312e8 P f6141944f9d7444dae7291f7e93a0db5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:41:53.122049 8352 raft_consensus.cc:2258] T 0ce81cdde3ab410c99b6d898f85312e8 P b354f0c3d7444c3281180585aeed4c5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:41:53.122303 8353 tablet_replica.cc:330] T 0ce81cdde3ab410c99b6d898f85312e8 P aee666d8594f44b09b3d8f73784f3979: stopping tablet replica
I20260620 18:41:53.122514 8353 raft_consensus.cc:2229] T 0ce81cdde3ab410c99b6d898f85312e8 P aee666d8594f44b09b3d8f73784f3979 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:41:53.122718 8353 raft_consensus.cc:2258] T 0ce81cdde3ab410c99b6d898f85312e8 P aee666d8594f44b09b3d8f73784f3979 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:41:53.122810 7763 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:41:53.123514 7763 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:41:53.124119 8352 ts_tablet_manager.cc:1905] T 0ce81cdde3ab410c99b6d898f85312e8 P b354f0c3d7444c3281180585aeed4c5b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:41:53.125716 8353 ts_tablet_manager.cc:1905] T 0ce81cdde3ab410c99b6d898f85312e8 P aee666d8594f44b09b3d8f73784f3979: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:41:53.125995 8354 ts_tablet_manager.cc:1905] T 0ce81cdde3ab410c99b6d898f85312e8 P f6141944f9d7444dae7291f7e93a0db5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:41:53.130896 7902 tablet_service.cc:1467] Processing CreateTablet for tablet a4a5ee8279dd4997abf974ed13879e02 (DEFAULT_TABLE table=test-table [id=2de41ddc1739414fa5e4851f1b2bb31d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:41:53.131320 7902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4a5ee8279dd4997abf974ed13879e02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:53.131609 8179 tablet_service.cc:1467] Processing CreateTablet for tablet a4a5ee8279dd4997abf974ed13879e02 (DEFAULT_TABLE table=test-table [id=2de41ddc1739414fa5e4851f1b2bb31d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:41:53.132009 8179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4a5ee8279dd4997abf974ed13879e02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:53.132028 8041 tablet_service.cc:1467] Processing CreateTablet for tablet a4a5ee8279dd4997abf974ed13879e02 (DEFAULT_TABLE table=test-table [id=2de41ddc1739414fa5e4851f1b2bb31d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:41:53.132433 8041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4a5ee8279dd4997abf974ed13879e02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:53.136220 8365 tablet_bootstrap.cc:492] T a4a5ee8279dd4997abf974ed13879e02 P f6141944f9d7444dae7291f7e93a0db5: Bootstrap starting.
I20260620 18:41:53.137269 8364 tablet_bootstrap.cc:492] T a4a5ee8279dd4997abf974ed13879e02 P b354f0c3d7444c3281180585aeed4c5b: Bootstrap starting.
I20260620 18:41:53.137460 8354 ts_tablet_manager.cc:1918] T 0ce81cdde3ab410c99b6d898f85312e8 P f6141944f9d7444dae7291f7e93a0db5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:41:53.137596 8354 log.cc:1196] T 0ce81cdde3ab410c99b6d898f85312e8 P f6141944f9d7444dae7291f7e93a0db5: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-2/wal/wals/0ce81cdde3ab410c99b6d898f85312e8
I20260620 18:41:53.137698 8353 ts_tablet_manager.cc:1918] T 0ce81cdde3ab410c99b6d898f85312e8 P aee666d8594f44b09b3d8f73784f3979: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:41:53.137836 8353 log.cc:1196] T 0ce81cdde3ab410c99b6d898f85312e8 P aee666d8594f44b09b3d8f73784f3979: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-1/wal/wals/0ce81cdde3ab410c99b6d898f85312e8
I20260620 18:41:53.137877 8365 tablet_bootstrap.cc:654] T a4a5ee8279dd4997abf974ed13879e02 P f6141944f9d7444dae7291f7e93a0db5: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:53.138077 8354 ts_tablet_manager.cc:1939] T 0ce81cdde3ab410c99b6d898f85312e8 P f6141944f9d7444dae7291f7e93a0db5: Deleting consensus metadata
I20260620 18:41:53.138298 8353 ts_tablet_manager.cc:1939] T 0ce81cdde3ab410c99b6d898f85312e8 P aee666d8594f44b09b3d8f73784f3979: Deleting consensus metadata
I20260620 18:41:53.138907 8364 tablet_bootstrap.cc:654] T a4a5ee8279dd4997abf974ed13879e02 P b354f0c3d7444c3281180585aeed4c5b: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:53.139122 7749 catalog_manager.cc:4841] TS f6141944f9d7444dae7291f7e93a0db5 (127.5.225.131:46387): tablet 0ce81cdde3ab410c99b6d898f85312e8 (table test-table [id=21938f2a37a3466ab23aa8598d3e1e17]) successfully deleted
I20260620 18:41:53.139330 7752 catalog_manager.cc:4841] TS aee666d8594f44b09b3d8f73784f3979 (127.5.225.130:42107): tablet 0ce81cdde3ab410c99b6d898f85312e8 (table test-table [id=21938f2a37a3466ab23aa8598d3e1e17]) successfully deleted
I20260620 18:41:53.141772 8364 tablet_bootstrap.cc:492] T a4a5ee8279dd4997abf974ed13879e02 P b354f0c3d7444c3281180585aeed4c5b: No bootstrap required, opened a new log
I20260620 18:41:53.141920 8364 ts_tablet_manager.cc:1397] T a4a5ee8279dd4997abf974ed13879e02 P b354f0c3d7444c3281180585aeed4c5b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 18:41:53.142287 8352 ts_tablet_manager.cc:1918] T 0ce81cdde3ab410c99b6d898f85312e8 P b354f0c3d7444c3281180585aeed4c5b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:41:53.142498 8352 log.cc:1196] T 0ce81cdde3ab410c99b6d898f85312e8 P b354f0c3d7444c3281180585aeed4c5b: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-0/wal/wals/0ce81cdde3ab410c99b6d898f85312e8
I20260620 18:41:53.142660 8363 tablet_bootstrap.cc:492] T a4a5ee8279dd4997abf974ed13879e02 P aee666d8594f44b09b3d8f73784f3979: Bootstrap starting.
I20260620 18:41:53.142985 8352 ts_tablet_manager.cc:1939] T 0ce81cdde3ab410c99b6d898f85312e8 P b354f0c3d7444c3281180585aeed4c5b: Deleting consensus metadata
I20260620 18:41:53.143579 8365 tablet_bootstrap.cc:492] T a4a5ee8279dd4997abf974ed13879e02 P f6141944f9d7444dae7291f7e93a0db5: No bootstrap required, opened a new log
I20260620 18:41:53.143718 8365 ts_tablet_manager.cc:1397] T a4a5ee8279dd4997abf974ed13879e02 P f6141944f9d7444dae7291f7e93a0db5: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260620 18:41:53.143848 8363 tablet_bootstrap.cc:654] T a4a5ee8279dd4997abf974ed13879e02 P aee666d8594f44b09b3d8f73784f3979: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:53.143991 7751 catalog_manager.cc:4841] TS b354f0c3d7444c3281180585aeed4c5b (127.5.225.129:37425): tablet 0ce81cdde3ab410c99b6d898f85312e8 (table test-table [id=21938f2a37a3466ab23aa8598d3e1e17]) successfully deleted
I20260620 18:41:53.144110 8364 raft_consensus.cc:348] T a4a5ee8279dd4997abf974ed13879e02 P b354f0c3d7444c3281180585aeed4c5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } }
I20260620 18:41:53.144336 8364 raft_consensus.cc:374] T a4a5ee8279dd4997abf974ed13879e02 P b354f0c3d7444c3281180585aeed4c5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:53.144426 8364 raft_consensus.cc:729] T a4a5ee8279dd4997abf974ed13879e02 P b354f0c3d7444c3281180585aeed4c5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b354f0c3d7444c3281180585aeed4c5b, State: Initialized, Role: FOLLOWER
I20260620 18:41:53.144394 8365 raft_consensus.cc:348] T a4a5ee8279dd4997abf974ed13879e02 P f6141944f9d7444dae7291f7e93a0db5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } }
I20260620 18:41:53.144646 8364 consensus_queue.cc:260] T a4a5ee8279dd4997abf974ed13879e02 P b354f0c3d7444c3281180585aeed4c5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } }
I20260620 18:41:53.144824 8365 raft_consensus.cc:374] T a4a5ee8279dd4997abf974ed13879e02 P f6141944f9d7444dae7291f7e93a0db5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:53.145169 8364 ts_tablet_manager.cc:1428] T a4a5ee8279dd4997abf974ed13879e02 P b354f0c3d7444c3281180585aeed4c5b: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20260620 18:41:53.145557 8365 raft_consensus.cc:729] T a4a5ee8279dd4997abf974ed13879e02 P f6141944f9d7444dae7291f7e93a0db5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6141944f9d7444dae7291f7e93a0db5, State: Initialized, Role: FOLLOWER
I20260620 18:41:53.145828 8365 consensus_queue.cc:260] T a4a5ee8279dd4997abf974ed13879e02 P f6141944f9d7444dae7291f7e93a0db5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } }
I20260620 18:41:53.146301 8365 ts_tablet_manager.cc:1428] T a4a5ee8279dd4997abf974ed13879e02 P f6141944f9d7444dae7291f7e93a0db5: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260620 18:41:53.147697 8363 tablet_bootstrap.cc:492] T a4a5ee8279dd4997abf974ed13879e02 P aee666d8594f44b09b3d8f73784f3979: No bootstrap required, opened a new log
I20260620 18:41:53.147869 8363 ts_tablet_manager.cc:1397] T a4a5ee8279dd4997abf974ed13879e02 P aee666d8594f44b09b3d8f73784f3979: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260620 18:41:53.148665 8363 raft_consensus.cc:348] T a4a5ee8279dd4997abf974ed13879e02 P aee666d8594f44b09b3d8f73784f3979 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } }
I20260620 18:41:53.148984 8363 raft_consensus.cc:374] T a4a5ee8279dd4997abf974ed13879e02 P aee666d8594f44b09b3d8f73784f3979 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:53.149130 8363 raft_consensus.cc:729] T a4a5ee8279dd4997abf974ed13879e02 P aee666d8594f44b09b3d8f73784f3979 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aee666d8594f44b09b3d8f73784f3979, State: Initialized, Role: FOLLOWER
I20260620 18:41:53.149463 8363 consensus_queue.cc:260] T a4a5ee8279dd4997abf974ed13879e02 P aee666d8594f44b09b3d8f73784f3979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } }
I20260620 18:41:53.150110 8363 ts_tablet_manager.cc:1428] T a4a5ee8279dd4997abf974ed13879e02 P aee666d8594f44b09b3d8f73784f3979: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 18:41:53.550881 8319 raft_consensus.cc:482] T a4a5ee8279dd4997abf974ed13879e02 P b354f0c3d7444c3281180585aeed4c5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:41:53.551092 8319 raft_consensus.cc:504] T a4a5ee8279dd4997abf974ed13879e02 P b354f0c3d7444c3281180585aeed4c5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } }
I20260620 18:41:53.551846 8319 leader_election.cc:288] T a4a5ee8279dd4997abf974ed13879e02 P b354f0c3d7444c3281180585aeed4c5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aee666d8594f44b09b3d8f73784f3979 (127.5.225.130:42107), f6141944f9d7444dae7291f7e93a0db5 (127.5.225.131:46387)
I20260620 18:41:53.552354 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4a5ee8279dd4997abf974ed13879e02" candidate_uuid: "b354f0c3d7444c3281180585aeed4c5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aee666d8594f44b09b3d8f73784f3979" is_pre_election: true
I20260620 18:41:53.552408 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4a5ee8279dd4997abf974ed13879e02" candidate_uuid: "b354f0c3d7444c3281180585aeed4c5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6141944f9d7444dae7291f7e93a0db5" is_pre_election: true
I20260620 18:41:53.552631 8061 raft_consensus.cc:2454] T a4a5ee8279dd4997abf974ed13879e02 P aee666d8594f44b09b3d8f73784f3979 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b354f0c3d7444c3281180585aeed4c5b in term 0.
I20260620 18:41:53.552757 8201 raft_consensus.cc:2454] T a4a5ee8279dd4997abf974ed13879e02 P f6141944f9d7444dae7291f7e93a0db5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b354f0c3d7444c3281180585aeed4c5b in term 0.
I20260620 18:41:53.553146 7855 leader_election.cc:302] T a4a5ee8279dd4997abf974ed13879e02 P b354f0c3d7444c3281180585aeed4c5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aee666d8594f44b09b3d8f73784f3979, b354f0c3d7444c3281180585aeed4c5b; no voters:
I20260620 18:41:53.553452 8319 raft_consensus.cc:2781] T a4a5ee8279dd4997abf974ed13879e02 P b354f0c3d7444c3281180585aeed4c5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:41:53.553553 8319 raft_consensus.cc:482] T a4a5ee8279dd4997abf974ed13879e02 P b354f0c3d7444c3281180585aeed4c5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:41:53.553618 8319 raft_consensus.cc:3037] T a4a5ee8279dd4997abf974ed13879e02 P b354f0c3d7444c3281180585aeed4c5b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:53.555732 8319 raft_consensus.cc:504] T a4a5ee8279dd4997abf974ed13879e02 P b354f0c3d7444c3281180585aeed4c5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } }
I20260620 18:41:53.556293 8319 leader_election.cc:288] T a4a5ee8279dd4997abf974ed13879e02 P b354f0c3d7444c3281180585aeed4c5b [CANDIDATE]: Term 1 election: Requested vote from peers aee666d8594f44b09b3d8f73784f3979 (127.5.225.130:42107), f6141944f9d7444dae7291f7e93a0db5 (127.5.225.131:46387)
I20260620 18:41:53.556730 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4a5ee8279dd4997abf974ed13879e02" candidate_uuid: "b354f0c3d7444c3281180585aeed4c5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aee666d8594f44b09b3d8f73784f3979"
I20260620 18:41:53.556756 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4a5ee8279dd4997abf974ed13879e02" candidate_uuid: "b354f0c3d7444c3281180585aeed4c5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6141944f9d7444dae7291f7e93a0db5"
I20260620 18:41:53.556938 8061 raft_consensus.cc:3037] T a4a5ee8279dd4997abf974ed13879e02 P aee666d8594f44b09b3d8f73784f3979 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:53.556964 8201 raft_consensus.cc:3037] T a4a5ee8279dd4997abf974ed13879e02 P f6141944f9d7444dae7291f7e93a0db5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:53.558877 8201 raft_consensus.cc:2454] T a4a5ee8279dd4997abf974ed13879e02 P f6141944f9d7444dae7291f7e93a0db5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b354f0c3d7444c3281180585aeed4c5b in term 1.
I20260620 18:41:53.559252 8061 raft_consensus.cc:2454] T a4a5ee8279dd4997abf974ed13879e02 P aee666d8594f44b09b3d8f73784f3979 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b354f0c3d7444c3281180585aeed4c5b in term 1.
I20260620 18:41:53.559283 7852 leader_election.cc:302] T a4a5ee8279dd4997abf974ed13879e02 P b354f0c3d7444c3281180585aeed4c5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b354f0c3d7444c3281180585aeed4c5b, f6141944f9d7444dae7291f7e93a0db5; no voters:
I20260620 18:41:53.559619 8319 raft_consensus.cc:2781] T a4a5ee8279dd4997abf974ed13879e02 P b354f0c3d7444c3281180585aeed4c5b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:41:53.559763 8319 raft_consensus.cc:686] T a4a5ee8279dd4997abf974ed13879e02 P b354f0c3d7444c3281180585aeed4c5b [term 1 LEADER]: Becoming Leader. State: Replica: b354f0c3d7444c3281180585aeed4c5b, State: Running, Role: LEADER
I20260620 18:41:53.560027 8319 consensus_queue.cc:237] T a4a5ee8279dd4997abf974ed13879e02 P b354f0c3d7444c3281180585aeed4c5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } }
I20260620 18:41:53.562528 7764 catalog_manager.cc:5495] T a4a5ee8279dd4997abf974ed13879e02 P b354f0c3d7444c3281180585aeed4c5b reported cstate change: term changed from 0 to 1, leader changed from <none> to b354f0c3d7444c3281180585aeed4c5b (127.5.225.129). New cstate: current_term: 1 leader_uuid: "b354f0c3d7444c3281180585aeed4c5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b354f0c3d7444c3281180585aeed4c5b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 } health_report { overall_health: UNKNOWN } } }
I20260620 18:41:53.623534 8061 raft_consensus.cc:1261] T a4a5ee8279dd4997abf974ed13879e02 P aee666d8594f44b09b3d8f73784f3979 [term 1 FOLLOWER]: Refusing update from remote peer b354f0c3d7444c3281180585aeed4c5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:41:53.623637 8201 raft_consensus.cc:1261] T a4a5ee8279dd4997abf974ed13879e02 P f6141944f9d7444dae7291f7e93a0db5 [term 1 FOLLOWER]: Refusing update from remote peer b354f0c3d7444c3281180585aeed4c5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:41:53.624092 8342 consensus_queue.cc:1035] T a4a5ee8279dd4997abf974ed13879e02 P b354f0c3d7444c3281180585aeed4c5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "aee666d8594f44b09b3d8f73784f3979" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 42107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:41:53.624886 8319 consensus_queue.cc:1035] T a4a5ee8279dd4997abf974ed13879e02 P b354f0c3d7444c3281180585aeed4c5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6141944f9d7444dae7291f7e93a0db5" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 18:41:53.644866 8335 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 18:41:55.234355 7765 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:41:55.234612 7765 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:41:55.238857 7765 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6100e6394dd4f48838ad1e248e2f501: Sending DeleteTablet for 3 replicas of tablet a4a5ee8279dd4997abf974ed13879e02
I20260620 18:41:55.239711 7902 tablet_service.cc:1514] Processing DeleteTablet for tablet a4a5ee8279dd4997abf974ed13879e02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:41:55 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:37966
I20260620 18:41:55.239754 8041 tablet_service.cc:1514] Processing DeleteTablet for tablet a4a5ee8279dd4997abf974ed13879e02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:41:55 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:53390
I20260620 18:41:55.239812 8179 tablet_service.cc:1514] Processing DeleteTablet for tablet a4a5ee8279dd4997abf974ed13879e02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:41:55 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:42656
I20260620 18:41:55.240378 8412 tablet_replica.cc:330] T a4a5ee8279dd4997abf974ed13879e02 P b354f0c3d7444c3281180585aeed4c5b: stopping tablet replica
I20260620 18:41:55.240628 8412 raft_consensus.cc:2229] T a4a5ee8279dd4997abf974ed13879e02 P b354f0c3d7444c3281180585aeed4c5b [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:41:55.240890 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 7836
I20260620 18:41:55.241093 8412 raft_consensus.cc:2258] T a4a5ee8279dd4997abf974ed13879e02 P b354f0c3d7444c3281180585aeed4c5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:41:55.241724 8411 tablet_replica.cc:330] T a4a5ee8279dd4997abf974ed13879e02 P aee666d8594f44b09b3d8f73784f3979: stopping tablet replica
I20260620 18:41:55.242043 8411 raft_consensus.cc:2229] T a4a5ee8279dd4997abf974ed13879e02 P aee666d8594f44b09b3d8f73784f3979 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:41:55.242296 8412 ts_tablet_manager.cc:1905] T a4a5ee8279dd4997abf974ed13879e02 P b354f0c3d7444c3281180585aeed4c5b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:41:55.242378 8411 raft_consensus.cc:2258] T a4a5ee8279dd4997abf974ed13879e02 P aee666d8594f44b09b3d8f73784f3979 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:41:55.243956 8411 ts_tablet_manager.cc:1905] T a4a5ee8279dd4997abf974ed13879e02 P aee666d8594f44b09b3d8f73784f3979: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:41:55.251394 8412 ts_tablet_manager.cc:1918] T a4a5ee8279dd4997abf974ed13879e02 P b354f0c3d7444c3281180585aeed4c5b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:41:55.251603 8412 log.cc:1196] T a4a5ee8279dd4997abf974ed13879e02 P b354f0c3d7444c3281180585aeed4c5b: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-0/wal/wals/a4a5ee8279dd4997abf974ed13879e02
I20260620 18:41:55.252195 8412 ts_tablet_manager.cc:1939] T a4a5ee8279dd4997abf974ed13879e02 P b354f0c3d7444c3281180585aeed4c5b: Deleting consensus metadata
I20260620 18:41:55.252683 8411 ts_tablet_manager.cc:1918] T a4a5ee8279dd4997abf974ed13879e02 P aee666d8594f44b09b3d8f73784f3979: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:41:55.252908 8411 log.cc:1196] T a4a5ee8279dd4997abf974ed13879e02 P aee666d8594f44b09b3d8f73784f3979: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-1/wal/wals/a4a5ee8279dd4997abf974ed13879e02
I20260620 18:41:55.253468 8411 ts_tablet_manager.cc:1939] T a4a5ee8279dd4997abf974ed13879e02 P aee666d8594f44b09b3d8f73784f3979: Deleting consensus metadata
I20260620 18:41:55.253558 7751 catalog_manager.cc:4841] TS b354f0c3d7444c3281180585aeed4c5b (127.5.225.129:37425): tablet a4a5ee8279dd4997abf974ed13879e02 (table test-table [id=2de41ddc1739414fa5e4851f1b2bb31d]) successfully deleted
I20260620 18:41:55.254629 8413 tablet_replica.cc:330] T a4a5ee8279dd4997abf974ed13879e02 P f6141944f9d7444dae7291f7e93a0db5: stopping tablet replica
I20260620 18:41:55.254720 7752 catalog_manager.cc:4841] TS aee666d8594f44b09b3d8f73784f3979 (127.5.225.130:42107): tablet a4a5ee8279dd4997abf974ed13879e02 (table test-table [id=2de41ddc1739414fa5e4851f1b2bb31d]) successfully deleted
I20260620 18:41:55.254901 8413 raft_consensus.cc:2229] T a4a5ee8279dd4997abf974ed13879e02 P f6141944f9d7444dae7291f7e93a0db5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:41:55.255169 8413 raft_consensus.cc:2258] T a4a5ee8279dd4997abf974ed13879e02 P f6141944f9d7444dae7291f7e93a0db5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:41:55.256187 8413 ts_tablet_manager.cc:1905] T a4a5ee8279dd4997abf974ed13879e02 P f6141944f9d7444dae7291f7e93a0db5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:41:55.261627 8413 ts_tablet_manager.cc:1918] T a4a5ee8279dd4997abf974ed13879e02 P f6141944f9d7444dae7291f7e93a0db5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:41:55.261737 8413 log.cc:1196] T a4a5ee8279dd4997abf974ed13879e02 P f6141944f9d7444dae7291f7e93a0db5: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980894474392-6022-0/minicluster-data/ts-2/wal/wals/a4a5ee8279dd4997abf974ed13879e02
I20260620 18:41:55.262029 8413 ts_tablet_manager.cc:1939] T a4a5ee8279dd4997abf974ed13879e02 P f6141944f9d7444dae7291f7e93a0db5: Deleting consensus metadata
I20260620 18:41:55.262941 7749 catalog_manager.cc:4841] TS f6141944f9d7444dae7291f7e93a0db5 (127.5.225.131:46387): tablet a4a5ee8279dd4997abf974ed13879e02 (table test-table [id=2de41ddc1739414fa5e4851f1b2bb31d]) successfully deleted
Jun 20 18:41:55 dist-test-slave-k6lg krb5kdc[7720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980912, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.225.129@KRBTEST.COM
I20260620 18:41:55.272827 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:55.241770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37992 (local address 127.5.225.129:37425)
0620 18:41:55.253942 (+ 12172us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:55.253947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:55.254641 (+ 694us) server_negotiation.cc:407] Connection header received
0620 18:41:55.254706 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:55.254711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:55.254783 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:55.254881 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:55.256713 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:55.257466 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:55.259297 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:55.259777 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:55.259880 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:55.267447 (+ 7567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:55.267464 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:55.267480 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:55.267521 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:55.270321 (+ 2800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:55.270980 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:55.270988 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:55.270990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:55.271112 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:55.271476 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:55.271481 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:55.271483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:55.271913 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:55.272137 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:55.272370 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 18:41:55.272533 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12067,"thread_start_us":11941,"threads_started":1}
W20260620 18:41:55.273499 7967 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37992
W20260620 18:41:55.274395 6022 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 7836: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:41:55.274616 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 7836
I20260620 18:41:55.297389 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 7976
Jun 20 18:41:55 dist-test-slave-k6lg krb5kdc[7720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980912, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.225.130@KRBTEST.COM
I20260620 18:41:55.312707 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:55.298170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53414 (local address 127.5.225.130:42107)
0620 18:41:55.298711 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:55.298715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:55.298732 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:41:55.298783 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:55.298787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:55.298847 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:55.298938 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:55.300158 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:55.300894 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:55.302056 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:55.302562 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:55.302697 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:55.306903 (+ 4206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:55.306923 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:55.306939 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:55.306973 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:55.309815 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:55.310615 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:55.310625 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:55.310629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:55.310781 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:55.311142 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:55.311147 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:55.311151 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:55.311526 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:55.311838 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:55.312130 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 18:41:55.312360 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":245,"threads_started":1}
W20260620 18:41:55.313351 8106 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53414
W20260620 18:41:55.313946 6022 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 7976: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:41:55.314072 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 7976
I20260620 18:41:55.335870 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 8115
Jun 20 18:41:55 dist-test-slave-k6lg krb5kdc[7720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980912, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.225.131@KRBTEST.COM
I20260620 18:41:55.352167 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:55.336664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42660 (local address 127.5.225.131:46387)
0620 18:41:55.337306 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:55.337310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:55.337327 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:41:55.337381 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:55.337385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:55.337442 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:55.337536 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:55.339145 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:55.339909 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:55.341282 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:55.341736 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:55.341841 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:55.346358 (+ 4517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:55.346375 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:55.346392 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:55.346430 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:55.349430 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:55.350250 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:55.350263 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:55.350269 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:55.350481 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:55.350861 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:55.350865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:55.350867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:55.351187 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:55.351401 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:55.351707 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 18:41:55.351886 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":368,"threads_started":1}
W20260620 18:41:55.352856 8245 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42660
W20260620 18:41:55.353440 6022 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 8115: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:41:55.353583 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 8115
I20260620 18:41:55.370910 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 7733
I20260620 18:41:55.372061 7805 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:55.492395 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 7733
2026-06-20T18:41:55Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8294 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:41:55 dist-test-slave-k6lg krb5kdc[8421](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:41:55 dist-test-slave-k6lg krb5kdc[8421](info): set up 2 sockets
Jun 20 18:41:55 dist-test-slave-k6lg krb5kdc[8421](info): commencing operation
krb5kdc: starting...
W20260620 18:41:56.594374 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.002s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:41:56 dist-test-slave-k6lg krb5kdc[8421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:41:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:41:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
I20260620 18:41:56.773396 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:45849
--webserver_interface=127.5.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:41997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:45849
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:57.090512 8434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:57.090865 8434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:57.091037 8434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:57.098217 8434 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:41:57.098379 8434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:57.098493 8434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:57.098608 8434 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:41:57.098695 8434 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:41:57.108462 8434 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:41997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:45849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:45849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:41:57.109865 8434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:57.111332 8434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:57.123540 8442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:57.123565 8440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:57.123616 8439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:57.124099 8434 server_base.cc:1034] running on GCE node
I20260620 18:41:57.124965 8434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:57.126515 8434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:57.127791 8434 hybrid_clock.cc:648] HybridClock initialized: now 1781980917127729 us; error 51 us; skew 500 ppm
I20260620 18:41:57.128130 8434 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:57 dist-test-slave-k6lg krb5kdc[8421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980917, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:57.131290 8434 init.cc:377] Logged in from keytab as kudu/127.5.225.190@KRBTEST.COM (short username kudu)
I20260620 18:41:57.132829 8434 webserver.cc:460] Webserver started at http://127.5.225.190:34167/ using document root <none> and password file <none>
I20260620 18:41:57.133509 8434 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:57.133662 8434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:57.133968 8434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:57.137027 8434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "246ed7b6ac5442cf8c922dc0e01ee811"
format_stamp: "Formatted at 2026-06-20 18:41:57 on dist-test-slave-k6lg"
server_key: "7ac608c22fae5852ddd2db35a5517926"
server_key_iv: "ffa6b85993bc607d3dbf30f3d844332a"
server_key_version: "encryptionkey@0"
I20260620 18:41:57.138103 8434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "246ed7b6ac5442cf8c922dc0e01ee811"
format_stamp: "Formatted at 2026-06-20 18:41:57 on dist-test-slave-k6lg"
server_key: "7ac608c22fae5852ddd2db35a5517926"
server_key_iv: "ffa6b85993bc607d3dbf30f3d844332a"
server_key_version: "encryptionkey@0"
I20260620 18:41:57.144333 8434 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 18:41:57.147771 8449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:57.149085 8434 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:41:57.149252 8434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "246ed7b6ac5442cf8c922dc0e01ee811"
format_stamp: "Formatted at 2026-06-20 18:41:57 on dist-test-slave-k6lg"
server_key: "7ac608c22fae5852ddd2db35a5517926"
server_key_iv: "ffa6b85993bc607d3dbf30f3d844332a"
server_key_version: "encryptionkey@0"
I20260620 18:41:57.149466 8434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:57.181164 8434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:57.185498 8434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:57.185837 8434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:57.204741 8434 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:45849
I20260620 18:41:57.204762 8500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:45849 every 8 connection(s)
I20260620 18:41:57.206266 8434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:41:57.210129 8501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:57.210148 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 8434
I20260620 18:41:57.210427 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:41:57.211226 6022 external_mini_cluster.cc:1428] Setting key 50ec22e805847278f7f8f11f8f7b530c
I20260620 18:41:57.224202 8501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 246ed7b6ac5442cf8c922dc0e01ee811: Bootstrap starting.
Jun 20 18:41:57 dist-test-slave-k6lg krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:41:57.228005 8501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 246ed7b6ac5442cf8c922dc0e01ee811: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:57.229437 8501 log.cc:824] T 00000000000000000000000000000000 P 246ed7b6ac5442cf8c922dc0e01ee811: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:57.232836 8501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 246ed7b6ac5442cf8c922dc0e01ee811: No bootstrap required, opened a new log
I20260620 18:41:57.236207 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:57.213195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34954 (local address 127.5.225.190:45849)
0620 18:41:57.214000 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:57.214011 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:57.214052 (+ 41us) server_negotiation.cc:407] Connection header received
0620 18:41:57.216275 (+ 2223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:57.216301 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:57.216727 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:57.217080 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:57.218986 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:57.220301 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:57.222213 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:57.222913 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:57.223056 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:57.227685 (+ 4629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:57.227697 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:57.227718 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:57.227771 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:57.231584 (+ 3813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:57.232285 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:57.232295 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:57.232305 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:57.232521 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:57.232929 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:57.232934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:57.232937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:57.233656 (+ 719us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:57.233950 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:57.234338 (+ 388us) server_negotiation.cc:299] Negotiation successful
0620 18:41:57.234813 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":390,"threads_started":1}
I20260620 18:41:57.241885 8501 raft_consensus.cc:348] T 00000000000000000000000000000000 P 246ed7b6ac5442cf8c922dc0e01ee811 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "246ed7b6ac5442cf8c922dc0e01ee811" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 45849 } }
I20260620 18:41:57.242276 8501 raft_consensus.cc:374] T 00000000000000000000000000000000 P 246ed7b6ac5442cf8c922dc0e01ee811 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:57.242354 8501 raft_consensus.cc:729] T 00000000000000000000000000000000 P 246ed7b6ac5442cf8c922dc0e01ee811 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 246ed7b6ac5442cf8c922dc0e01ee811, State: Initialized, Role: FOLLOWER
I20260620 18:41:57.242888 8501 consensus_queue.cc:260] T 00000000000000000000000000000000 P 246ed7b6ac5442cf8c922dc0e01ee811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "246ed7b6ac5442cf8c922dc0e01ee811" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 45849 } }
I20260620 18:41:57.243129 8501 raft_consensus.cc:388] T 00000000000000000000000000000000 P 246ed7b6ac5442cf8c922dc0e01ee811 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:41:57.243260 8501 raft_consensus.cc:482] T 00000000000000000000000000000000 P 246ed7b6ac5442cf8c922dc0e01ee811 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:41:57.243428 8501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 246ed7b6ac5442cf8c922dc0e01ee811 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:57.245774 8501 raft_consensus.cc:504] T 00000000000000000000000000000000 P 246ed7b6ac5442cf8c922dc0e01ee811 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "246ed7b6ac5442cf8c922dc0e01ee811" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 45849 } }
I20260620 18:41:57.246302 8501 leader_election.cc:302] T 00000000000000000000000000000000 P 246ed7b6ac5442cf8c922dc0e01ee811 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 246ed7b6ac5442cf8c922dc0e01ee811; no voters:
I20260620 18:41:57.246870 8501 leader_election.cc:288] T 00000000000000000000000000000000 P 246ed7b6ac5442cf8c922dc0e01ee811 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:41:57.247013 8506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 246ed7b6ac5442cf8c922dc0e01ee811 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:41:57.248206 8506 raft_consensus.cc:686] T 00000000000000000000000000000000 P 246ed7b6ac5442cf8c922dc0e01ee811 [term 1 LEADER]: Becoming Leader. State: Replica: 246ed7b6ac5442cf8c922dc0e01ee811, State: Running, Role: LEADER
I20260620 18:41:57.248792 8506 consensus_queue.cc:237] T 00000000000000000000000000000000 P 246ed7b6ac5442cf8c922dc0e01ee811 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "246ed7b6ac5442cf8c922dc0e01ee811" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 45849 } }
I20260620 18:41:57.249122 8501 sys_catalog.cc:564] T 00000000000000000000000000000000 P 246ed7b6ac5442cf8c922dc0e01ee811 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:41:57.251463 8508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 246ed7b6ac5442cf8c922dc0e01ee811 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 246ed7b6ac5442cf8c922dc0e01ee811. Latest consensus state: current_term: 1 leader_uuid: "246ed7b6ac5442cf8c922dc0e01ee811" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "246ed7b6ac5442cf8c922dc0e01ee811" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 45849 } } }
I20260620 18:41:57.251817 8508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 246ed7b6ac5442cf8c922dc0e01ee811 [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:57.252322 8507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 246ed7b6ac5442cf8c922dc0e01ee811 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "246ed7b6ac5442cf8c922dc0e01ee811" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "246ed7b6ac5442cf8c922dc0e01ee811" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 45849 } } }
I20260620 18:41:57.252600 8507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 246ed7b6ac5442cf8c922dc0e01ee811 [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:57.252976 8514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:41:57.258495 8514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:41:57.267297 8514 catalog_manager.cc:1347] Generated new cluster ID: 5a0b3c03a5d34ac1b8f11148e2662963
I20260620 18:41:57.267427 8514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:41:57.283113 8514 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:41:57.285017 8514 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:41:57.292811 8514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 246ed7b6ac5442cf8c922dc0e01ee811: Generated new TSK 0
I20260620 18:41:57.293889 8514 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab.
Entry for principal kudu/127.5.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab.
I20260620 18:41:57.341652 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.129:0
--local_ip_for_outbound_sockets=127.5.225.129
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:45849
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:41997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:57.666612 8527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:57.666916 8527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:57.667088 8527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:57.674202 8527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:57.674358 8527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:57.674530 8527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.129
I20260620 18:41:57.683784 8527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:41997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:45849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.129
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:41:57.685001 8527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:57.686302 8527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:57.699296 8532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:57.699285 8535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:57.699280 8533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:57.700460 8527 server_base.cc:1034] running on GCE node
I20260620 18:41:57.701061 8527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:57.701959 8527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:57.703152 8527 hybrid_clock.cc:648] HybridClock initialized: now 1781980917703102 us; error 48 us; skew 500 ppm
I20260620 18:41:57.703461 8527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:57 dist-test-slave-k6lg krb5kdc[8421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980917, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:57.706539 8527 init.cc:377] Logged in from keytab as kudu/127.5.225.129@KRBTEST.COM (short username kudu)
I20260620 18:41:57.707988 8527 webserver.cc:460] Webserver started at http://127.5.225.129:36309/ using document root <none> and password file <none>
I20260620 18:41:57.708657 8527 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:57.708796 8527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:57.709103 8527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:57.712100 8527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-0/data/instance:
uuid: "a75bd393611948d686d16d324c6aa9f0"
format_stamp: "Formatted at 2026-06-20 18:41:57 on dist-test-slave-k6lg"
server_key: "b8763dd20ec6f2c3c2e001973e002c0b"
server_key_iv: "1bbf8367245a9a70acff6808b86d381f"
server_key_version: "encryptionkey@0"
I20260620 18:41:57.712864 8527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance:
uuid: "a75bd393611948d686d16d324c6aa9f0"
format_stamp: "Formatted at 2026-06-20 18:41:57 on dist-test-slave-k6lg"
server_key: "b8763dd20ec6f2c3c2e001973e002c0b"
server_key_iv: "1bbf8367245a9a70acff6808b86d381f"
server_key_version: "encryptionkey@0"
I20260620 18:41:57.717782 8527 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:41:57.721203 8542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:57.722515 8527 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:41:57.722689 8527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-0/wal
uuid: "a75bd393611948d686d16d324c6aa9f0"
format_stamp: "Formatted at 2026-06-20 18:41:57 on dist-test-slave-k6lg"
server_key: "b8763dd20ec6f2c3c2e001973e002c0b"
server_key_iv: "1bbf8367245a9a70acff6808b86d381f"
server_key_version: "encryptionkey@0"
I20260620 18:41:57.722889 8527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:57.752456 8527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:57.756939 8527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:57.757242 8527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:57.758256 8527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:41:57.759532 8527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:57.759632 8527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:57.759776 8527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:57.759855 8527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:57.792519 8527 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.129:34309
I20260620 18:41:57.792620 8654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.129:34309 every 8 connection(s)
I20260620 18:41:57.793876 8527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
I20260620 18:41:57.801388 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 8527
I20260620 18:41:57.801648 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance
I20260620 18:41:57.802335 6022 external_mini_cluster.cc:1428] Setting key 925c17f824ecd8e9e8ca2bbd142a0621
Jun 20 18:41:57 dist-test-slave-k6lg krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980917, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:41:57.814889 8657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:57.796490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:43849 (local address 127.5.225.190:45849)
0620 18:41:57.796898 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:57.796902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:57.797863 (+ 961us) server_negotiation.cc:407] Connection header received
0620 18:41:57.799701 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:57.799707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:57.799799 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:57.799916 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:57.801715 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:57.802598 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:57.804410 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:57.804798 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:57.804945 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:57.808395 (+ 3450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:57.808400 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:57.808404 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:57.808436 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:57.811554 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:57.812371 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:57.812379 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:57.812386 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:57.812541 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:57.812933 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:57.812941 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:57.812947 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:57.813306 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:57.813472 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:57.814130 (+ 658us) server_negotiation.cc:299] Negotiation successful
0620 18:41:57.814399 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":199,"threads_started":1}
I20260620 18:41:57.815529 8658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:57.797060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45849 (local address 127.5.225.129:43849)
0620 18:41:57.797707 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:57.797743 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:57.799427 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:57.800149 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:57.800157 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:57.800888 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:57.801506 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:57.801525 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:57.802754 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:57.802761 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:57.804269 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:57.804281 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:57.805020 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:57.805028 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:57.805276 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:57.805994 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:57.806016 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:57.808188 (+ 2172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:57.811694 (+ 3506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:57.811701 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:57.811714 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:57.812211 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:57.812644 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:57.812648 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:57.812650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:57.812824 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:57.813438 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:57.813444 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:57.813761 (+ 317us) client_negotiation.cc:769] Sending connection context
0620 18:41:57.813985 (+ 224us) client_negotiation.cc:240] Negotiation successful
0620 18:41:57.814426 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":261,"threads_started":1}
I20260620 18:41:57.817067 8655 heartbeater.cc:347] Connected to a master server at 127.5.225.190:45849
I20260620 18:41:57.817410 8655 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:57.818392 8655 heartbeater.cc:511] Master 127.5.225.190:45849 requested a full tablet report, sending...
I20260620 18:41:57.820582 8466 ts_manager.cc:194] Registered new tserver with Master: a75bd393611948d686d16d324c6aa9f0 (127.5.225.129:34309)
WARNING: no policy specified for kudu/127.5.225.130@KRBTEST.COM; defaulting to no policy
I20260620 18:41:57.822095 8466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.129@KRBTEST.COM'} at 127.5.225.129:43849
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab.
Entry for principal kudu/127.5.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab.
I20260620 18:41:57.845641 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.130:0
--local_ip_for_outbound_sockets=127.5.225.130
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:45849
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:41997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:58.161155 8661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:58.161549 8661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:58.161725 8661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:58.168987 8661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:58.169132 8661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:58.169320 8661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.130
I20260620 18:41:58.179191 8661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:41997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:45849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.130
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:41:58.180519 8661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:58.181914 8661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:58.194928 8666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:58.195007 8667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:58.194959 8669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:58.199745 8661 server_base.cc:1034] running on GCE node
I20260620 18:41:58.200381 8661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:58.201334 8661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:58.202533 8661 hybrid_clock.cc:648] HybridClock initialized: now 1781980918202473 us; error 50 us; skew 500 ppm
I20260620 18:41:58.202862 8661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:58 dist-test-slave-k6lg krb5kdc[8421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980918, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:58.206080 8661 init.cc:377] Logged in from keytab as kudu/127.5.225.130@KRBTEST.COM (short username kudu)
I20260620 18:41:58.207676 8661 webserver.cc:460] Webserver started at http://127.5.225.130:36945/ using document root <none> and password file <none>
I20260620 18:41:58.208372 8661 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:58.208492 8661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:58.208804 8661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:58.211799 8661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-1/data/instance:
uuid: "b773cfaf9d0f4423bf3416e719f6bf56"
format_stamp: "Formatted at 2026-06-20 18:41:58 on dist-test-slave-k6lg"
server_key: "437436493fe0b9bb0c6173d6c931b14c"
server_key_iv: "64fd13c2649f8659594a98b2546685fb"
server_key_version: "encryptionkey@0"
I20260620 18:41:58.212598 8661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance:
uuid: "b773cfaf9d0f4423bf3416e719f6bf56"
format_stamp: "Formatted at 2026-06-20 18:41:58 on dist-test-slave-k6lg"
server_key: "437436493fe0b9bb0c6173d6c931b14c"
server_key_iv: "64fd13c2649f8659594a98b2546685fb"
server_key_version: "encryptionkey@0"
I20260620 18:41:58.218278 8661 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 18:41:58.221774 8676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:58.223111 8661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:41:58.223276 8661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-1/wal
uuid: "b773cfaf9d0f4423bf3416e719f6bf56"
format_stamp: "Formatted at 2026-06-20 18:41:58 on dist-test-slave-k6lg"
server_key: "437436493fe0b9bb0c6173d6c931b14c"
server_key_iv: "64fd13c2649f8659594a98b2546685fb"
server_key_version: "encryptionkey@0"
I20260620 18:41:58.223470 8661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:58.245267 8661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:58.249157 8661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:58.249445 8661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:58.250398 8661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:41:58.251710 8661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:58.251811 8661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:58.251959 8661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:58.252030 8661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:58.285682 8661 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.130:36785
I20260620 18:41:58.285784 8788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.130:36785 every 8 connection(s)
I20260620 18:41:58.287055 8661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
I20260620 18:41:58.293687 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 8661
I20260620 18:41:58.293941 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance
I20260620 18:41:58.294461 6022 external_mini_cluster.cc:1428] Setting key 695e1c6315ca9391264b59fce31b9b66
Jun 20 18:41:58 dist-test-slave-k6lg krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980918, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:41:58.309458 8657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:58.289837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:41975 (local address 127.5.225.190:45849)
0620 18:41:58.290060 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:58.290067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:58.291317 (+ 1250us) server_negotiation.cc:407] Connection header received
0620 18:41:58.293126 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:58.293131 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:58.293218 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:58.293328 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:58.295120 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:58.296091 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:58.297662 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:58.298470 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:58.298627 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:58.302331 (+ 3704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:58.302336 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:58.302339 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:58.302370 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:58.306068 (+ 3698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:58.306854 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:58.306862 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:58.306868 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:58.307027 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:58.307427 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:58.307434 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:58.307440 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:58.307824 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:58.307982 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:58.308700 (+ 718us) server_negotiation.cc:299] Negotiation successful
0620 18:41:58.308957 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":84}
I20260620 18:41:58.310225 8791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:58.290440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45849 (local address 127.5.225.130:41975)
0620 18:41:58.291153 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:58.291190 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:58.292843 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:58.293585 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:58.293593 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:58.294304 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:58.294924 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:58.294942 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:58.296249 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:58.296254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:58.297503 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:58.297515 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:58.298675 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:58.298680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:58.298866 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:58.299561 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:58.299586 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:58.302036 (+ 2450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:58.306261 (+ 4225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:58.306268 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:58.306282 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:58.306714 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:58.307138 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:58.307141 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:58.307143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:58.307311 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:58.307944 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:58.307949 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:58.308258 (+ 309us) client_negotiation.cc:769] Sending connection context
0620 18:41:58.308509 (+ 251us) client_negotiation.cc:240] Negotiation successful
0620 18:41:58.308970 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":275,"threads_started":1}
I20260620 18:41:58.311733 8789 heartbeater.cc:347] Connected to a master server at 127.5.225.190:45849
I20260620 18:41:58.312081 8789 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:58.312990 8789 heartbeater.cc:511] Master 127.5.225.190:45849 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.225.131@KRBTEST.COM; defaulting to no policy
I20260620 18:41:58.314670 8466 ts_manager.cc:194] Registered new tserver with Master: b773cfaf9d0f4423bf3416e719f6bf56 (127.5.225.130:36785)
I20260620 18:41:58.315514 8466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.130@KRBTEST.COM'} at 127.5.225.130:41975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab.
Entry for principal kudu/127.5.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab.
I20260620 18:41:58.337599 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.131:0
--local_ip_for_outbound_sockets=127.5.225.131
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:45849
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:41997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:58.667508 8794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:58.667788 8794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:58.667954 8794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:58.675721 8794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:58.675837 8794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:58.676035 8794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.131
I20260620 18:41:58.686120 8794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:41997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:45849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.131
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:41:58.687572 8794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:58.688896 8794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:58.704555 8799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:58.704432 8800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:58.705595 8802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:58.705755 8794 server_base.cc:1034] running on GCE node
I20260620 18:41:58.706485 8794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:58.707557 8794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:58.708815 8794 hybrid_clock.cc:648] HybridClock initialized: now 1781980918708739 us; error 57 us; skew 500 ppm
I20260620 18:41:58.709244 8794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:58 dist-test-slave-k6lg krb5kdc[8421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980918, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:58.712584 8794 init.cc:377] Logged in from keytab as kudu/127.5.225.131@KRBTEST.COM (short username kudu)
I20260620 18:41:58.714372 8794 webserver.cc:460] Webserver started at http://127.5.225.131:40469/ using document root <none> and password file <none>
I20260620 18:41:58.715176 8794 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:58.715317 8794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:58.715718 8794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:58.718861 8794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-2/data/instance:
uuid: "3dafc0cd39fe41aebfacb4737a4197ae"
format_stamp: "Formatted at 2026-06-20 18:41:58 on dist-test-slave-k6lg"
server_key: "ed13ae0b08f0adfb16e57807137eaee4"
server_key_iv: "32452d1a943200905ecfd45f1a9f6f2f"
server_key_version: "encryptionkey@0"
I20260620 18:41:58.719650 8794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance:
uuid: "3dafc0cd39fe41aebfacb4737a4197ae"
format_stamp: "Formatted at 2026-06-20 18:41:58 on dist-test-slave-k6lg"
server_key: "ed13ae0b08f0adfb16e57807137eaee4"
server_key_iv: "32452d1a943200905ecfd45f1a9f6f2f"
server_key_version: "encryptionkey@0"
I20260620 18:41:58.725392 8794 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:41:58.729563 8809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:58.731117 8794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:41:58.731282 8794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-2/wal
uuid: "3dafc0cd39fe41aebfacb4737a4197ae"
format_stamp: "Formatted at 2026-06-20 18:41:58 on dist-test-slave-k6lg"
server_key: "ed13ae0b08f0adfb16e57807137eaee4"
server_key_iv: "32452d1a943200905ecfd45f1a9f6f2f"
server_key_version: "encryptionkey@0"
I20260620 18:41:58.731477 8794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:58.744861 8794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:58.749568 8794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:58.749861 8794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:58.750988 8794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:41:58.752506 8794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:58.752604 8794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:58.752753 8794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:58.752827 8794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:58.790323 8794 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.131:34937
I20260620 18:41:58.790346 8921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.131:34937 every 8 connection(s)
I20260620 18:41:58.791663 8794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
I20260620 18:41:58.796671 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 8794
I20260620 18:41:58.796882 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance
I20260620 18:41:58.797513 6022 external_mini_cluster.cc:1428] Setting key c739842122da87d13ccf522d395484ce
Jun 20 18:41:58 dist-test-slave-k6lg krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980918, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:41:58.824723 8657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:58.796126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:41665 (local address 127.5.225.190:45849)
0620 18:41:58.798256 (+ 2130us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:58.798265 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:58.806900 (+ 8635us) server_negotiation.cc:407] Connection header received
0620 18:41:58.808743 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:58.808749 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:58.808864 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:58.808992 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:58.810579 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:58.811567 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:58.813651 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:58.814263 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:58.814378 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:58.818823 (+ 4445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:58.818828 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:58.818832 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:58.818862 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:58.821602 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:58.822400 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:58.822410 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:58.822412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:58.822548 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:58.822996 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:58.823003 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:58.823011 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:58.823272 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:58.823435 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:58.824129 (+ 694us) server_negotiation.cc:299] Negotiation successful
0620 18:41:58.824343 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1923}
I20260620 18:41:58.825188 8655 heartbeater.cc:503] Master 127.5.225.190:45849 was elected leader, sending a full tablet report...
I20260620 18:41:58.826220 8924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:58.795472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45849 (local address 127.5.225.131:41665)
0620 18:41:58.806736 (+ 11264us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:58.806778 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:58.808474 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:58.809225 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:58.809233 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:58.809783 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:58.810410 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:58.810429 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:58.811725 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:58.811729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:58.813482 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:58.813499 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:58.814535 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:58.814545 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:58.814934 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:58.815711 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:58.815747 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:58.818620 (+ 2873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:58.821753 (+ 3133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:58.821762 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:58.821779 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:58.822277 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:58.822663 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:58.822668 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:58.822672 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:58.822877 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:58.823401 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:58.823409 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:58.823751 (+ 342us) client_negotiation.cc:769] Sending connection context
0620 18:41:58.824016 (+ 265us) client_negotiation.cc:240] Negotiation successful
0620 18:41:58.824535 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10938,"thread_start_us":10724,"threads_started":1}
I20260620 18:41:58.828033 8922 heartbeater.cc:347] Connected to a master server at 127.5.225.190:45849
I20260620 18:41:58.828351 8922 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:58.829051 8922 heartbeater.cc:511] Master 127.5.225.190:45849 requested a full tablet report, sending...
I20260620 18:41:58.830611 8466 ts_manager.cc:194] Registered new tserver with Master: 3dafc0cd39fe41aebfacb4737a4197ae (127.5.225.131:34937)
I20260620 18:41:58.831424 8466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.131@KRBTEST.COM'} at 127.5.225.131:41665
I20260620 18:41:58.840468 6022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 18:41:58.871340 8657 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:41:58.868662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34960 (local address 127.5.225.190:45849)
0620 18:41:58.868834 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:58.868838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:58.869333 (+ 495us) server_negotiation.cc:407] Connection header received
0620 18:41:58.869568 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:58.869574 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:58.869651 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:58.869765 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:58.871091 (+ 1326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34960: BlockingRecv error: recv got EOF from 127.0.0.1:34960 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:41:58.875030 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 8527
W20260620 18:41:58.884246 6022 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 8527: Not authorized: Client connection negotiation failed: client connection to 127.5.225.129:34309: server requires authentication, but client does not have Kerberos credentials available
I20260620 18:41:58.884475 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 8527
I20260620 18:41:58.900451 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 8661
W20260620 18:41:58.903463 6022 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 8661: Not authorized: Client connection negotiation failed: client connection to 127.5.225.130:36785: server requires authentication, but client does not have Kerberos credentials available
I20260620 18:41:58.903648 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 8661
I20260620 18:41:58.927217 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 8794
W20260620 18:41:58.930866 6022 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 8794: Not authorized: Client connection negotiation failed: client connection to 127.5.225.131:34937: server requires authentication, but client does not have Kerberos credentials available
I20260620 18:41:58.931015 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 8794
I20260620 18:41:58.943826 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 8434
I20260620 18:41:58.945036 8496 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:59.049350 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 8434
2026-06-20T18:41:59Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3553 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:41:59 dist-test-slave-k6lg krb5kdc[8939](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:41:59 dist-test-slave-k6lg krb5kdc[8939](info): set up 2 sockets
Jun 20 18:41:59 dist-test-slave-k6lg krb5kdc[8939](info): commencing operation
krb5kdc: starting...
W20260620 18:42:00.156106 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260620 18:42:00.396816 6022 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.241s user 0.000s sys 0.005s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260620 18:42:00.599197 6022 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.202s user 0.000s sys 0.005s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260620 18:42:00.861446 6022 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.262s user 0.000s sys 0.004s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980894474392-6022-0/krb5kdc/test-user.keytab.
W20260620 18:42:01.109874 6022 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.248s user 0.000s sys 0.005s
Jun 20 18:42:01 dist-test-slave-k6lg krb5kdc[8939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:42:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:42:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
I20260620 18:42:01.199355 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:36331
--webserver_interface=127.5.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:41509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:36331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:01.518926 8952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:01.519280 8952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:01.519448 8952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:01.526866 8952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:42:01.527014 8952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:01.527124 8952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:01.527230 8952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:42:01.527323 8952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:42:01.537401 8952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:41509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:36331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:36331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:01.538895 8952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:01.540333 8952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:01.552263 8957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:01.552340 8960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:01.552377 8958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:01.553437 8952 server_base.cc:1034] running on GCE node
I20260620 18:42:01.554315 8952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:01.555763 8952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:01.557019 8952 hybrid_clock.cc:648] HybridClock initialized: now 1781980921556972 us; error 35 us; skew 500 ppm
I20260620 18:42:01.557411 8952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:01 dist-test-slave-k6lg krb5kdc[8939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980921, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:01.564680 8952 init.cc:377] Logged in from keytab as kudu/127.5.225.190@KRBTEST.COM (short username kudu)
I20260620 18:42:01.566488 8952 webserver.cc:460] Webserver started at http://127.5.225.190:43987/ using document root <none> and password file <none>
I20260620 18:42:01.567219 8952 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:01.567356 8952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:01.567689 8952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:01.570751 8952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "c4ebd8846c91481a91fc25cab619d2de"
format_stamp: "Formatted at 2026-06-20 18:42:01 on dist-test-slave-k6lg"
server_key: "6dd611d2d9e5a601e29257fa4ba00752"
server_key_iv: "98ab6b06960e91561ad022e8982652aa"
server_key_version: "encryptionkey@0"
I20260620 18:42:01.571533 8952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "c4ebd8846c91481a91fc25cab619d2de"
format_stamp: "Formatted at 2026-06-20 18:42:01 on dist-test-slave-k6lg"
server_key: "6dd611d2d9e5a601e29257fa4ba00752"
server_key_iv: "98ab6b06960e91561ad022e8982652aa"
server_key_version: "encryptionkey@0"
I20260620 18:42:01.576682 8952 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:42:01.580030 8967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:01.581333 8952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:42:01.581511 8952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "c4ebd8846c91481a91fc25cab619d2de"
format_stamp: "Formatted at 2026-06-20 18:42:01 on dist-test-slave-k6lg"
server_key: "6dd611d2d9e5a601e29257fa4ba00752"
server_key_iv: "98ab6b06960e91561ad022e8982652aa"
server_key_version: "encryptionkey@0"
I20260620 18:42:01.581708 8952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:01.595227 8952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:01.599124 8952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:01.599407 8952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:01.618119 8952 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:36331
I20260620 18:42:01.618191 9018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:36331 every 8 connection(s)
I20260620 18:42:01.619570 8952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:42:01.623355 9019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:01.627460 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 8952
I20260620 18:42:01.627682 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:42:01.628475 6022 external_mini_cluster.cc:1428] Setting key 47fc3bf8f3cf8c2bc8b87dd0618a2d78
I20260620 18:42:01.637154 9019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4ebd8846c91481a91fc25cab619d2de: Bootstrap starting.
I20260620 18:42:01.641086 9019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4ebd8846c91481a91fc25cab619d2de: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:01.642580 9019 log.cc:824] T 00000000000000000000000000000000 P c4ebd8846c91481a91fc25cab619d2de: Log is configured to *not* fsync() on all Append() calls
Jun 20 18:42:01 dist-test-slave-k6lg krb5kdc[8939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:42:01.645606 9019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4ebd8846c91481a91fc25cab619d2de: No bootstrap required, opened a new log
I20260620 18:42:01.652309 9022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:01.630568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51672 (local address 127.5.225.190:36331)
0620 18:42:01.631345 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:01.631357 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:01.631396 (+ 39us) server_negotiation.cc:407] Connection header received
0620 18:42:01.632912 (+ 1516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:01.632933 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:01.633342 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:01.633679 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:01.635622 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:01.636941 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:01.638731 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:01.639418 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:01.639571 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:01.644493 (+ 4922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:01.644502 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:01.644517 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:01.644556 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:01.647860 (+ 3304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:01.648657 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:01.648664 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:01.648674 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:01.648847 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:01.649266 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:01.649271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:01.649273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:01.649909 (+ 636us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:01.650275 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:01.650781 (+ 506us) server_negotiation.cc:299] Negotiation successful
0620 18:42:01.651222 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":380,"threads_started":1}
I20260620 18:42:01.654057 9019 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4ebd8846c91481a91fc25cab619d2de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4ebd8846c91481a91fc25cab619d2de" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36331 } }
I20260620 18:42:01.654522 9019 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4ebd8846c91481a91fc25cab619d2de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:01.654649 9019 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4ebd8846c91481a91fc25cab619d2de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4ebd8846c91481a91fc25cab619d2de, State: Initialized, Role: FOLLOWER
I20260620 18:42:01.655333 9019 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4ebd8846c91481a91fc25cab619d2de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4ebd8846c91481a91fc25cab619d2de" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36331 } }
I20260620 18:42:01.655575 9019 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4ebd8846c91481a91fc25cab619d2de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:42:01.655715 9019 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4ebd8846c91481a91fc25cab619d2de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:42:01.655911 9019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4ebd8846c91481a91fc25cab619d2de [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:01.656625 6022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:42:01.658246 9019 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4ebd8846c91481a91fc25cab619d2de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4ebd8846c91481a91fc25cab619d2de" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36331 } }
I20260620 18:42:01.658780 9019 leader_election.cc:302] T 00000000000000000000000000000000 P c4ebd8846c91481a91fc25cab619d2de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4ebd8846c91481a91fc25cab619d2de; no voters:
I20260620 18:42:01.659409 9019 leader_election.cc:288] T 00000000000000000000000000000000 P c4ebd8846c91481a91fc25cab619d2de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:42:01.659561 9024 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4ebd8846c91481a91fc25cab619d2de [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:42:01.661655 9019 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4ebd8846c91481a91fc25cab619d2de [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:42:01.662432 9024 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4ebd8846c91481a91fc25cab619d2de [term 1 LEADER]: Becoming Leader. State: Replica: c4ebd8846c91481a91fc25cab619d2de, State: Running, Role: LEADER
I20260620 18:42:01.663118 9024 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4ebd8846c91481a91fc25cab619d2de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4ebd8846c91481a91fc25cab619d2de" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36331 } }
I20260620 18:42:01.668494 9025 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4ebd8846c91481a91fc25cab619d2de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4ebd8846c91481a91fc25cab619d2de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4ebd8846c91481a91fc25cab619d2de" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36331 } } }
I20260620 18:42:01.668470 9026 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4ebd8846c91481a91fc25cab619d2de [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4ebd8846c91481a91fc25cab619d2de. Latest consensus state: current_term: 1 leader_uuid: "c4ebd8846c91481a91fc25cab619d2de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4ebd8846c91481a91fc25cab619d2de" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36331 } } }
I20260620 18:42:01.668972 9025 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4ebd8846c91481a91fc25cab619d2de [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:01.669015 9026 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4ebd8846c91481a91fc25cab619d2de [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:01.679302 9034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:42:01.685350 9034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:42:01.705953 9034 catalog_manager.cc:1347] Generated new cluster ID: c91cc262264f490f805194a709e25a75
I20260620 18:42:01.706089 9034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:42:01.713898 9022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:01.706104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51686 (local address 127.5.225.190:36331)
0620 18:42:01.706310 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:01.706318 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:01.707254 (+ 936us) server_negotiation.cc:407] Connection header received
0620 18:42:01.707485 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:01.707490 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:01.707555 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:01.707672 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:01.708649 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:01.709365 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:01.710710 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:01.711137 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:01.711268 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:01.712396 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:01.712402 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:01.712406 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:42:01.713076 (+ 670us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:01.713146 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:42:01.713174 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:01.713292 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:01.713433 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 18:42:01.713532 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 18:42:01.735061 9034 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:42:01.736984 9034 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:42:01.752756 9034 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4ebd8846c91481a91fc25cab619d2de: Generated new TSK 0
I20260620 18:42:01.753837 9034 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:42:01.764246 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 8952
I20260620 18:42:01.765583 9014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:01.856882 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 8952
2026-06-20T18:42:01Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2801 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:42:01 dist-test-slave-k6lg krb5kdc[9053](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:42:01 dist-test-slave-k6lg krb5kdc[9053](info): set up 2 sockets
Jun 20 18:42:01 dist-test-slave-k6lg krb5kdc[9053](info): commencing operation
krb5kdc: starting...
W20260620 18:42:02.930573 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:42:03 dist-test-slave-k6lg krb5kdc[9053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:42:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:42:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
I20260620 18:42:03.121153 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:44503
--webserver_interface=127.5.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:34303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:44503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:03.440188 9066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:03.440469 9066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:03.440672 9066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:03.447976 9066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:42:03.448089 9066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:03.448173 9066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:03.448275 9066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:42:03.448349 9066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:42:03.458354 9066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:34303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:44503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:44503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:03.459704 9066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:03.461061 9066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:03.473014 9074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:03.473047 9071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:03.473069 9072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:03.473701 9066 server_base.cc:1034] running on GCE node
I20260620 18:42:03.474535 9066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:03.475900 9066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:03.477133 9066 hybrid_clock.cc:648] HybridClock initialized: now 1781980923477093 us; error 38 us; skew 500 ppm
I20260620 18:42:03.477473 9066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:03 dist-test-slave-k6lg krb5kdc[9053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980923, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:03.480647 9066 init.cc:377] Logged in from keytab as kudu/127.5.225.190@KRBTEST.COM (short username kudu)
I20260620 18:42:03.482049 9066 webserver.cc:460] Webserver started at http://127.5.225.190:34815/ using document root <none> and password file <none>
I20260620 18:42:03.482730 9066 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:03.482875 9066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:03.483182 9066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:03.486022 9066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "d0d2bbb416bf4358b2ceb1237ca78ff8"
format_stamp: "Formatted at 2026-06-20 18:42:03 on dist-test-slave-k6lg"
server_key: "e7809c66613f09b8fa326d6ee6d9fc84"
server_key_iv: "4dab572c66143b736d9423104ca27545"
server_key_version: "encryptionkey@0"
I20260620 18:42:03.486832 9066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "d0d2bbb416bf4358b2ceb1237ca78ff8"
format_stamp: "Formatted at 2026-06-20 18:42:03 on dist-test-slave-k6lg"
server_key: "e7809c66613f09b8fa326d6ee6d9fc84"
server_key_iv: "4dab572c66143b736d9423104ca27545"
server_key_version: "encryptionkey@0"
I20260620 18:42:03.491942 9066 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:42:03.495213 9081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:03.496459 9066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:42:03.496639 9066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "d0d2bbb416bf4358b2ceb1237ca78ff8"
format_stamp: "Formatted at 2026-06-20 18:42:03 on dist-test-slave-k6lg"
server_key: "e7809c66613f09b8fa326d6ee6d9fc84"
server_key_iv: "4dab572c66143b736d9423104ca27545"
server_key_version: "encryptionkey@0"
I20260620 18:42:03.496824 9066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:03.520354 9066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:03.524798 9066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:03.525079 9066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:03.543555 9066 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:44503
I20260620 18:42:03.543586 9132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:44503 every 8 connection(s)
I20260620 18:42:03.544879 9066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:42:03.548524 9133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:03.549279 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 9066
I20260620 18:42:03.549511 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:42:03.550220 6022 external_mini_cluster.cc:1428] Setting key cdaab64c4b152392d0184744ccf3d6ae
I20260620 18:42:03.562072 9133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0d2bbb416bf4358b2ceb1237ca78ff8: Bootstrap starting.
Jun 20 18:42:03 dist-test-slave-k6lg krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:42:03.565698 9133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0d2bbb416bf4358b2ceb1237ca78ff8: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:03.566911 9133 log.cc:824] T 00000000000000000000000000000000 P d0d2bbb416bf4358b2ceb1237ca78ff8: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:03.569772 9133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0d2bbb416bf4358b2ceb1237ca78ff8: No bootstrap required, opened a new log
I20260620 18:42:03.573271 9136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:03.552264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57536 (local address 127.5.225.190:44503)
0620 18:42:03.553109 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:03.553121 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:03.553161 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:42:03.554731 (+ 1570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:03.554754 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:03.555143 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:03.555478 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:03.557137 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:03.558508 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:03.560319 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:03.560987 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:03.561143 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:03.565641 (+ 4498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:03.565649 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:03.565663 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:03.565705 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:03.569157 (+ 3452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:03.569819 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:03.569831 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:03.569837 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:03.570033 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:03.570443 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:03.570449 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:03.570452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:03.570943 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:03.571388 (+ 445us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:03.571874 (+ 486us) server_negotiation.cc:299] Negotiation successful
0620 18:42:03.572308 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":426,"threads_started":1}
I20260620 18:42:03.577419 9133 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0d2bbb416bf4358b2ceb1237ca78ff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d2bbb416bf4358b2ceb1237ca78ff8" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 44503 } }
I20260620 18:42:03.577755 9133 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0d2bbb416bf4358b2ceb1237ca78ff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:03.577862 9133 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0d2bbb416bf4358b2ceb1237ca78ff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0d2bbb416bf4358b2ceb1237ca78ff8, State: Initialized, Role: FOLLOWER
I20260620 18:42:03.578512 9133 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0d2bbb416bf4358b2ceb1237ca78ff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d2bbb416bf4358b2ceb1237ca78ff8" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 44503 } }
I20260620 18:42:03.578806 9133 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0d2bbb416bf4358b2ceb1237ca78ff8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:42:03.578938 9133 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0d2bbb416bf4358b2ceb1237ca78ff8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:42:03.579137 9133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0d2bbb416bf4358b2ceb1237ca78ff8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:03.581468 9133 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0d2bbb416bf4358b2ceb1237ca78ff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d2bbb416bf4358b2ceb1237ca78ff8" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 44503 } }
I20260620 18:42:03.581914 9133 leader_election.cc:302] T 00000000000000000000000000000000 P d0d2bbb416bf4358b2ceb1237ca78ff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0d2bbb416bf4358b2ceb1237ca78ff8; no voters:
I20260620 18:42:03.582564 9133 leader_election.cc:288] T 00000000000000000000000000000000 P d0d2bbb416bf4358b2ceb1237ca78ff8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:42:03.582693 9138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0d2bbb416bf4358b2ceb1237ca78ff8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:42:03.583377 9138 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0d2bbb416bf4358b2ceb1237ca78ff8 [term 1 LEADER]: Becoming Leader. State: Replica: d0d2bbb416bf4358b2ceb1237ca78ff8, State: Running, Role: LEADER
I20260620 18:42:03.583913 9138 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0d2bbb416bf4358b2ceb1237ca78ff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d2bbb416bf4358b2ceb1237ca78ff8" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 44503 } }
I20260620 18:42:03.584524 9133 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0d2bbb416bf4358b2ceb1237ca78ff8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:42:03.587119 9140 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0d2bbb416bf4358b2ceb1237ca78ff8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0d2bbb416bf4358b2ceb1237ca78ff8. Latest consensus state: current_term: 1 leader_uuid: "d0d2bbb416bf4358b2ceb1237ca78ff8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d2bbb416bf4358b2ceb1237ca78ff8" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 44503 } } }
I20260620 18:42:03.587556 9140 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0d2bbb416bf4358b2ceb1237ca78ff8 [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:03.587438 9139 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0d2bbb416bf4358b2ceb1237ca78ff8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0d2bbb416bf4358b2ceb1237ca78ff8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d2bbb416bf4358b2ceb1237ca78ff8" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 44503 } } }
I20260620 18:42:03.587757 9139 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0d2bbb416bf4358b2ceb1237ca78ff8 [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:03.589026 9147 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:42:03.593271 9147 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:42:03.601491 9147 catalog_manager.cc:1347] Generated new cluster ID: c42339c14cf84f5aae5fd76a2784248b
I20260620 18:42:03.601632 9147 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:42:03.616354 9147 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:42:03.617637 9147 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:42:03.629173 9147 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0d2bbb416bf4358b2ceb1237ca78ff8: Generated new TSK 0
I20260620 18:42:03.629974 9147 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab.
Entry for principal kudu/127.5.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab.
I20260620 18:42:03.675067 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.129:0
--local_ip_for_outbound_sockets=127.5.225.129
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:44503
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:34303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:04.014453 9159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:04.014726 9159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:04.014819 9159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:04.021981 9159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:04.022095 9159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:04.022285 9159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.129
I20260620 18:42:04.032166 9159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:34303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:44503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.129
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:04.033386 9159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:04.034650 9159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:04.047817 9164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:04.047784 9165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:04.047713 9167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:04.048911 9159 server_base.cc:1034] running on GCE node
I20260620 18:42:04.049470 9159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:04.050359 9159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:04.051554 9159 hybrid_clock.cc:648] HybridClock initialized: now 1781980924051492 us; error 53 us; skew 500 ppm
I20260620 18:42:04.051862 9159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:04 dist-test-slave-k6lg krb5kdc[9053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980924, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:04.054929 9159 init.cc:377] Logged in from keytab as kudu/127.5.225.129@KRBTEST.COM (short username kudu)
I20260620 18:42:04.056353 9159 webserver.cc:460] Webserver started at http://127.5.225.129:36553/ using document root <none> and password file <none>
I20260620 18:42:04.057035 9159 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:04.057163 9159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:04.057472 9159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:04.060334 9159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-0/data/instance:
uuid: "6ddeb60dd39a4f0eb343678791332823"
format_stamp: "Formatted at 2026-06-20 18:42:04 on dist-test-slave-k6lg"
server_key: "8ba61a456d92b49b803c78392ed31f3a"
server_key_iv: "fbfb8a77b226bd4c9fe510cef8e6cfe2"
server_key_version: "encryptionkey@0"
I20260620 18:42:04.061103 9159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance:
uuid: "6ddeb60dd39a4f0eb343678791332823"
format_stamp: "Formatted at 2026-06-20 18:42:04 on dist-test-slave-k6lg"
server_key: "8ba61a456d92b49b803c78392ed31f3a"
server_key_iv: "fbfb8a77b226bd4c9fe510cef8e6cfe2"
server_key_version: "encryptionkey@0"
I20260620 18:42:04.065976 9159 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260620 18:42:04.069345 9174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:04.070681 9159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:42:04.070840 9159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-0/wal
uuid: "6ddeb60dd39a4f0eb343678791332823"
format_stamp: "Formatted at 2026-06-20 18:42:04 on dist-test-slave-k6lg"
server_key: "8ba61a456d92b49b803c78392ed31f3a"
server_key_iv: "fbfb8a77b226bd4c9fe510cef8e6cfe2"
server_key_version: "encryptionkey@0"
I20260620 18:42:04.071017 9159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:04.081107 9159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:04.085119 9159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:04.085394 9159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:04.086380 9159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:04.087620 9159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:04.087692 9159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:04.087846 9159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:04.087906 9159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:04.121647 9159 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.129:39893
I20260620 18:42:04.121686 9286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.129:39893 every 8 connection(s)
I20260620 18:42:04.123009 9159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
I20260620 18:42:04.125854 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 9159
I20260620 18:42:04.126039 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance
I20260620 18:42:04.126907 6022 external_mini_cluster.cc:1428] Setting key a18c306f47b89eb1aa16521304f93510
Jun 20 18:42:04 dist-test-slave-k6lg krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980924, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.225.130@KRBTEST.COM; defaulting to no policy
I20260620 18:42:04.146601 9289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:04.125660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:51069 (local address 127.5.225.190:44503)
0620 18:42:04.126255 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:04.126260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:04.127287 (+ 1027us) server_negotiation.cc:407] Connection header received
0620 18:42:04.129591 (+ 2304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:04.129597 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:04.129679 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:04.129794 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:04.131383 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:04.132577 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:04.134486 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:04.135201 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:04.135356 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:04.139561 (+ 4205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:04.139570 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:04.139578 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:04.139624 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:04.143262 (+ 3638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:04.144036 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:04.144041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:04.144045 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:04.144190 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:04.144607 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:04.144614 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:04.144619 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:04.145004 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:04.145157 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:04.145893 (+ 736us) server_negotiation.cc:299] Negotiation successful
0620 18:42:04.146135 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":336,"threads_started":1}
I20260620 18:42:04.147378 9290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:04.126305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:44503 (local address 127.5.225.129:51069)
0620 18:42:04.127132 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:04.127170 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:04.129290 (+ 2120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:04.130033 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:04.130041 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:04.130593 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:04.131210 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:04.131237 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:04.132720 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:04.132724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:04.134287 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:04.134311 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:04.135480 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:04.135515 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:04.135897 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:04.136626 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:04.136660 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:04.139305 (+ 2645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:04.143423 (+ 4118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:04.143430 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:04.143444 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:04.143893 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:04.144308 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:04.144311 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:04.144313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:04.144479 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:04.145132 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:04.145137 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:04.145443 (+ 306us) client_negotiation.cc:769] Sending connection context
0620 18:42:04.145706 (+ 263us) client_negotiation.cc:240] Negotiation successful
0620 18:42:04.146113 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":316,"threads_started":1}
I20260620 18:42:04.148949 9287 heartbeater.cc:347] Connected to a master server at 127.5.225.190:44503
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.130@KRBTEST.COM" created.
I20260620 18:42:04.149273 9287 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:04.149974 9287 heartbeater.cc:511] Master 127.5.225.190:44503 requested a full tablet report, sending...
I20260620 18:42:04.152050 9098 ts_manager.cc:194] Registered new tserver with Master: 6ddeb60dd39a4f0eb343678791332823 (127.5.225.129:39893)
I20260620 18:42:04.153458 9098 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.129@KRBTEST.COM'} at 127.5.225.129:51069
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab.
Entry for principal kudu/127.5.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab.
I20260620 18:42:04.173111 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.130:0
--local_ip_for_outbound_sockets=127.5.225.130
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:44503
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:34303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:04.489120 9293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:04.489444 9293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:04.489653 9293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:04.496858 9293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:04.497004 9293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:04.497185 9293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.130
I20260620 18:42:04.506865 9293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:34303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:44503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.130
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:04.508188 9293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:04.509649 9293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:04.522424 9298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:04.522397 9301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:04.522514 9299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:04.523030 9293 server_base.cc:1034] running on GCE node
I20260620 18:42:04.523628 9293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:04.524773 9293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:04.526026 9293 hybrid_clock.cc:648] HybridClock initialized: now 1781980924525981 us; error 64 us; skew 500 ppm
I20260620 18:42:04.526401 9293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:04 dist-test-slave-k6lg krb5kdc[9053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980924, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:04.529682 9293 init.cc:377] Logged in from keytab as kudu/127.5.225.130@KRBTEST.COM (short username kudu)
I20260620 18:42:04.531164 9293 webserver.cc:460] Webserver started at http://127.5.225.130:46373/ using document root <none> and password file <none>
I20260620 18:42:04.531796 9293 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:04.531894 9293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:04.532140 9293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:04.534996 9293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-1/data/instance:
uuid: "b27dadcda77849ae9082b798f5677bce"
format_stamp: "Formatted at 2026-06-20 18:42:04 on dist-test-slave-k6lg"
server_key: "4c1c60909e3a1569558b32323532faa9"
server_key_iv: "9d8c74fcaa934397f050372cdfff49bb"
server_key_version: "encryptionkey@0"
I20260620 18:42:04.535717 9293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance:
uuid: "b27dadcda77849ae9082b798f5677bce"
format_stamp: "Formatted at 2026-06-20 18:42:04 on dist-test-slave-k6lg"
server_key: "4c1c60909e3a1569558b32323532faa9"
server_key_iv: "9d8c74fcaa934397f050372cdfff49bb"
server_key_version: "encryptionkey@0"
I20260620 18:42:04.540671 9293 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:42:04.544222 9308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:04.545681 9293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 18:42:04.545836 9293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-1/wal
uuid: "b27dadcda77849ae9082b798f5677bce"
format_stamp: "Formatted at 2026-06-20 18:42:04 on dist-test-slave-k6lg"
server_key: "4c1c60909e3a1569558b32323532faa9"
server_key_iv: "9d8c74fcaa934397f050372cdfff49bb"
server_key_version: "encryptionkey@0"
I20260620 18:42:04.546012 9293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:04.591329 9293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:04.595724 9293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:04.596056 9293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:04.597090 9293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:04.598459 9293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:04.598538 9293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:04.598655 9293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:04.598717 9293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:04.631858 9293 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.130:35341
I20260620 18:42:04.631903 9420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.130:35341 every 8 connection(s)
I20260620 18:42:04.633131 9293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
I20260620 18:42:04.640748 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 9293
I20260620 18:42:04.640905 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance
I20260620 18:42:04.641427 6022 external_mini_cluster.cc:1428] Setting key 66364abab4103f437fa118181f18d083
Jun 20 18:42:04 dist-test-slave-k6lg krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980924, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:42:04.653765 9289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:04.635576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:56755 (local address 127.5.225.190:44503)
0620 18:42:04.635788 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:04.635794 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:04.636933 (+ 1139us) server_negotiation.cc:407] Connection header received
0620 18:42:04.638729 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:04.638733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:04.638803 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:04.638892 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:04.640312 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:04.641269 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:04.643266 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:04.643756 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:04.643854 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:04.647948 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:04.647953 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:04.647956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:04.647988 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:04.650826 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:04.651539 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:04.651543 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:04.651545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:04.651658 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:04.652083 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:04.652088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:04.652091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:04.652336 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:04.652485 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:04.653159 (+ 674us) server_negotiation.cc:299] Negotiation successful
0620 18:42:04.653385 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 18:42:04.654793 9423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:04.636110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:44503 (local address 127.5.225.130:56755)
0620 18:42:04.636772 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:04.636814 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:04.638480 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:04.639098 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:04.639106 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:04.639581 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:04.640142 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:04.640160 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:04.641442 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:04.641451 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:04.643091 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:04.643104 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:04.643976 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:04.643983 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:04.644298 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:04.645034 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:04.645079 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:04.647720 (+ 2641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:04.650961 (+ 3241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:04.650970 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:04.650986 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:04.651401 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:04.651766 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:04.651771 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:04.651775 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:04.651960 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:04.652462 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:04.652469 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:04.652799 (+ 330us) client_negotiation.cc:769] Sending connection context
0620 18:42:04.653072 (+ 273us) client_negotiation.cc:240] Negotiation successful
0620 18:42:04.653608 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":259,"threads_started":1}
I20260620 18:42:04.656240 9421 heartbeater.cc:347] Connected to a master server at 127.5.225.190:44503
I20260620 18:42:04.656543 9421 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:04.657279 9421 heartbeater.cc:511] Master 127.5.225.190:44503 requested a full tablet report, sending...
I20260620 18:42:04.658761 9098 ts_manager.cc:194] Registered new tserver with Master: b27dadcda77849ae9082b798f5677bce (127.5.225.130:35341)
WARNING: no policy specified for kudu/127.5.225.131@KRBTEST.COM; defaulting to no policy
I20260620 18:42:04.659634 9098 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.130@KRBTEST.COM'} at 127.5.225.130:56755
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab.
Entry for principal kudu/127.5.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab.
I20260620 18:42:04.685480 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.131:0
--local_ip_for_outbound_sockets=127.5.225.131
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:44503
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:34303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:05.007273 9426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:05.007601 9426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:05.007699 9426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:05.015726 9426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:05.015864 9426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:05.015985 9426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.131
I20260620 18:42:05.025961 9426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:34303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:44503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.131
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:05.027451 9426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:05.028869 9426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:05.042403 9432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:05.042448 9431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:05.043018 9434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:05.047806 9426 server_base.cc:1034] running on GCE node
I20260620 18:42:05.048544 9426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:05.049712 9426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:05.050998 9426 hybrid_clock.cc:648] HybridClock initialized: now 1781980925050943 us; error 38 us; skew 500 ppm
I20260620 18:42:05.051349 9426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:05 dist-test-slave-k6lg krb5kdc[9053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980925, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:05.054780 9426 init.cc:377] Logged in from keytab as kudu/127.5.225.131@KRBTEST.COM (short username kudu)
I20260620 18:42:05.056361 9426 webserver.cc:460] Webserver started at http://127.5.225.131:34793/ using document root <none> and password file <none>
I20260620 18:42:05.057071 9426 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:05.057217 9426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:05.057529 9426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:05.060452 9426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-2/data/instance:
uuid: "dc1b4d73e85040aaac2d825003a7791b"
format_stamp: "Formatted at 2026-06-20 18:42:05 on dist-test-slave-k6lg"
server_key: "5462c5d4852eab28319b80b9e9c13583"
server_key_iv: "d4a5e1ffd6b2f25767e8dafe16df4e08"
server_key_version: "encryptionkey@0"
I20260620 18:42:05.061228 9426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance:
uuid: "dc1b4d73e85040aaac2d825003a7791b"
format_stamp: "Formatted at 2026-06-20 18:42:05 on dist-test-slave-k6lg"
server_key: "5462c5d4852eab28319b80b9e9c13583"
server_key_iv: "d4a5e1ffd6b2f25767e8dafe16df4e08"
server_key_version: "encryptionkey@0"
I20260620 18:42:05.066399 9426 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260620 18:42:05.069664 9441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:05.071071 9426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:42:05.071274 9426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-2/wal
uuid: "dc1b4d73e85040aaac2d825003a7791b"
format_stamp: "Formatted at 2026-06-20 18:42:05 on dist-test-slave-k6lg"
server_key: "5462c5d4852eab28319b80b9e9c13583"
server_key_iv: "d4a5e1ffd6b2f25767e8dafe16df4e08"
server_key_version: "encryptionkey@0"
I20260620 18:42:05.071482 9426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:05.085460 9426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:05.089424 9426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:05.089757 9426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:05.090900 9426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:05.092240 9426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:05.092342 9426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:05.092490 9426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:05.092618 9426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:05.126590 9426 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.131:45971
I20260620 18:42:05.126631 9553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.131:45971 every 8 connection(s)
I20260620 18:42:05.127787 9426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
I20260620 18:42:05.134126 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 9426
I20260620 18:42:05.134334 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance
I20260620 18:42:05.134883 6022 external_mini_cluster.cc:1428] Setting key 7e48effeaf0481021bb1aa93c3eb1fa9
Jun 20 18:42:05 dist-test-slave-k6lg krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980925, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:42:05.147248 9289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:05.130350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:40191 (local address 127.5.225.190:44503)
0620 18:42:05.130539 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:05.130544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:05.131593 (+ 1049us) server_negotiation.cc:407] Connection header received
0620 18:42:05.133389 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:05.133395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:05.133508 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:05.133619 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:05.135278 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:05.136351 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:05.138286 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:05.138694 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:05.138799 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:05.142091 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:05.142096 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:05.142099 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:05.142128 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:05.144592 (+ 2464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:05.145258 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:05.145262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:05.145264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:05.145388 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:05.145791 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:05.145794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:05.145796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:05.146055 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:05.146224 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:05.146854 (+ 630us) server_negotiation.cc:299] Negotiation successful
0620 18:42:05.147011 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 18:42:05.148185 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:05.130727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:44503 (local address 127.5.225.131:40191)
0620 18:42:05.131417 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:05.131468 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:05.133116 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:05.133844 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:05.133852 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:05.134496 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:05.135116 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:05.135136 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:05.136520 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:05.136525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:05.138105 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:05.138117 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:05.138880 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:05.138885 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:05.139091 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:05.139750 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:05.139772 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:05.141888 (+ 2116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:05.144720 (+ 2832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:05.144726 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:05.144737 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:05.145124 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:05.145480 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:05.145483 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:05.145486 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:05.145689 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:05.146197 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:05.146210 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:05.146489 (+ 279us) client_negotiation.cc:769] Sending connection context
0620 18:42:05.146740 (+ 251us) client_negotiation.cc:240] Negotiation successful
0620 18:42:05.147086 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":273,"threads_started":1}
I20260620 18:42:05.149570 9554 heartbeater.cc:347] Connected to a master server at 127.5.225.190:44503
I20260620 18:42:05.149864 9554 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:05.150663 9554 heartbeater.cc:511] Master 127.5.225.190:44503 requested a full tablet report, sending...
I20260620 18:42:05.152051 9098 ts_manager.cc:194] Registered new tserver with Master: dc1b4d73e85040aaac2d825003a7791b (127.5.225.131:45971)
I20260620 18:42:05.152783 9098 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.131@KRBTEST.COM'} at 127.5.225.131:40191
I20260620 18:42:05.156234 9287 heartbeater.cc:503] Master 127.5.225.190:44503 was elected leader, sending a full tablet report...
I20260620 18:42:05.162142 6022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 18:42:05 dist-test-slave-k6lg krb5kdc[9053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980925, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 18:42:05 dist-test-slave-k6lg krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980925, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:42:05.210050 9289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:05.196074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57544 (local address 127.5.225.190:44503)
0620 18:42:05.196263 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:05.196268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:05.196807 (+ 539us) server_negotiation.cc:407] Connection header received
0620 18:42:05.196943 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:05.196947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:05.197027 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:05.197138 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:05.198504 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:05.199241 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:05.200509 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:05.200920 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:05.201120 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:05.204960 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:05.204965 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:05.204968 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:05.205006 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:05.207502 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:05.208111 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:05.208123 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:05.208125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:05.208243 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:05.208640 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:05.208645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:05.208649 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:05.208967 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:05.209214 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:05.209519 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 18:42:05.209727 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 18:42:05.210652 9098 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57544
I20260620 18:42:05.212486 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 9159
Jun 20 18:42:05 dist-test-slave-k6lg krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980925, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.225.129@KRBTEST.COM
I20260620 18:42:05.228411 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:05.213474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44662 (local address 127.5.225.129:39893)
0620 18:42:05.214118 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:05.214125 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:05.214150 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:42:05.214261 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:05.214267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:05.214489 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:05.214619 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:05.215738 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:05.216768 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:05.218044 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:05.218699 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:05.218798 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:05.222690 (+ 3892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:05.222701 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:05.222716 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:05.222752 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:05.225518 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:05.226365 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:05.226373 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:05.226376 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:05.226531 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:05.226899 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:05.226902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:05.226904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:05.227233 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:05.227456 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:05.227775 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 18:42:05.227965 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":314,"threads_started":1}
W20260620 18:42:05.229741 9282 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44662
W20260620 18:42:05.230875 6022 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 9159: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:42:05.231063 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 9159
I20260620 18:42:05.246625 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 9293
Jun 20 18:42:05 dist-test-slave-k6lg krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980925, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.225.130@KRBTEST.COM
I20260620 18:42:05.263875 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:05.247547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49284 (local address 127.5.225.130:35341)
0620 18:42:05.248170 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:05.248177 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:05.248203 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:42:05.248264 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:05.248270 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:05.248483 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:05.248622 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:05.250007 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:05.251158 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:05.253085 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:05.253697 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:05.253796 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:05.258134 (+ 4338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:05.258145 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:05.258200 (+ 55us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:05.258241 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:05.261098 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:05.261761 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:05.261767 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:05.261770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:05.261882 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:05.262511 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:05.262514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:05.262516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:05.262848 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:05.263076 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:05.263359 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 18:42:05.263568 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":329,"threads_started":1}
W20260620 18:42:05.265210 9416 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49284
W20260620 18:42:05.266475 6022 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 9293: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:42:05.266631 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 9293
I20260620 18:42:05.282011 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 9426
Jun 20 18:42:05 dist-test-slave-k6lg krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980925, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.225.131@KRBTEST.COM
I20260620 18:42:05.299178 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:05.283242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36936 (local address 127.5.225.131:45971)
0620 18:42:05.283919 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:05.283926 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:05.283956 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:42:05.284021 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:05.284028 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:05.284324 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:05.284529 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:05.285908 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:05.287216 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:05.288575 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:05.289123 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:05.289218 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:05.293523 (+ 4305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:05.293534 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:05.293548 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:05.293579 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:05.296440 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:05.297145 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:05.297153 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:05.297155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:05.297281 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:05.297649 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:05.297653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:05.297655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:05.297991 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:05.298303 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:05.298671 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 18:42:05.298886 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":375,"threads_started":1}
W20260620 18:42:05.300580 9549 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36936
W20260620 18:42:05.301792 6022 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 9426: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:42:05.301990 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 9426
I20260620 18:42:05.317243 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 9066
I20260620 18:42:05.318519 9128 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:05.423913 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 9066
2026-06-20T18:42:05Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3572 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:42:05 dist-test-slave-k6lg krb5kdc[9571](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:42:05 dist-test-slave-k6lg krb5kdc[9571](info): set up 2 sockets
Jun 20 18:42:05 dist-test-slave-k6lg krb5kdc[9571](info): commencing operation
krb5kdc: starting...
W20260620 18:42:06.506547 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:42:06 dist-test-slave-k6lg krb5kdc[9571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:42:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:42:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
I20260620 18:42:06.695734 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:40021
--webserver_interface=127.5.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:36153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:40021
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:07.015033 9584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:07.015336 9584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:07.015498 9584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:07.023604 9584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:42:07.023736 9584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:07.023864 9584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:07.024015 9584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:42:07.024088 9584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:42:07.035540 9584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:36153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:40021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:40021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:07.037014 9584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:07.038497 9584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:07.051759 9592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:07.051697 9589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:07.051753 9590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:07.053032 9584 server_base.cc:1034] running on GCE node
I20260620 18:42:07.054028 9584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:07.055486 9584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:07.056741 9584 hybrid_clock.cc:648] HybridClock initialized: now 1781980927056697 us; error 67 us; skew 500 ppm
I20260620 18:42:07.057130 9584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:07 dist-test-slave-k6lg krb5kdc[9571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980927, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:07.063510 9584 init.cc:377] Logged in from keytab as kudu/127.5.225.190@KRBTEST.COM (short username kudu)
I20260620 18:42:07.065186 9584 webserver.cc:460] Webserver started at http://127.5.225.190:41245/ using document root <none> and password file <none>
I20260620 18:42:07.065912 9584 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:07.066061 9584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:07.066428 9584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:07.069427 9584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "907e7632d20342bcbf9d4bda55158857"
format_stamp: "Formatted at 2026-06-20 18:42:07 on dist-test-slave-k6lg"
server_key: "4cf2ce7b4dab62668e07c6d2e4774567"
server_key_iv: "2097fba930459a774debf93b760c957c"
server_key_version: "encryptionkey@0"
I20260620 18:42:07.070298 9584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "907e7632d20342bcbf9d4bda55158857"
format_stamp: "Formatted at 2026-06-20 18:42:07 on dist-test-slave-k6lg"
server_key: "4cf2ce7b4dab62668e07c6d2e4774567"
server_key_iv: "2097fba930459a774debf93b760c957c"
server_key_version: "encryptionkey@0"
I20260620 18:42:07.075552 9584 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260620 18:42:07.079381 9599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:07.080796 9584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:42:07.081000 9584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "907e7632d20342bcbf9d4bda55158857"
format_stamp: "Formatted at 2026-06-20 18:42:07 on dist-test-slave-k6lg"
server_key: "4cf2ce7b4dab62668e07c6d2e4774567"
server_key_iv: "2097fba930459a774debf93b760c957c"
server_key_version: "encryptionkey@0"
I20260620 18:42:07.081212 9584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:07.104432 9584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:07.109184 9584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:07.109498 9584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:07.128003 9584 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:40021
I20260620 18:42:07.128059 9650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:40021 every 8 connection(s)
I20260620 18:42:07.129405 9584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:42:07.133356 9651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:07.133927 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 9584
I20260620 18:42:07.134183 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:42:07.135008 6022 external_mini_cluster.cc:1428] Setting key 66d8e4516781484ca42decf8ce5d6f4d
I20260620 18:42:07.147771 9651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 907e7632d20342bcbf9d4bda55158857: Bootstrap starting.
Jun 20 18:42:07 dist-test-slave-k6lg krb5kdc[9571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:42:07.151556 9651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 907e7632d20342bcbf9d4bda55158857: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:07.152806 9651 log.cc:824] T 00000000000000000000000000000000 P 907e7632d20342bcbf9d4bda55158857: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:07.155896 9651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 907e7632d20342bcbf9d4bda55158857: No bootstrap required, opened a new log
I20260620 18:42:07.158802 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:07.137172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50358 (local address 127.5.225.190:40021)
0620 18:42:07.137999 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:07.138011 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:07.138056 (+ 45us) server_negotiation.cc:407] Connection header received
0620 18:42:07.139906 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:07.139931 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:07.140331 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:07.140675 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:07.142560 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:07.143840 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:07.145714 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:07.146512 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:07.146659 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:07.151272 (+ 4613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:07.151281 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:07.151296 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:07.151340 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:07.154592 (+ 3252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:07.155313 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:07.155323 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:07.155332 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:07.155534 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:07.155973 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:07.155978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:07.155981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:07.156553 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:07.156864 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:07.157274 (+ 410us) server_negotiation.cc:299] Negotiation successful
0620 18:42:07.157727 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":376,"threads_started":1}
I20260620 18:42:07.164340 9651 raft_consensus.cc:348] T 00000000000000000000000000000000 P 907e7632d20342bcbf9d4bda55158857 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907e7632d20342bcbf9d4bda55158857" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 40021 } }
I20260620 18:42:07.164716 9651 raft_consensus.cc:374] T 00000000000000000000000000000000 P 907e7632d20342bcbf9d4bda55158857 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:07.164829 9651 raft_consensus.cc:729] T 00000000000000000000000000000000 P 907e7632d20342bcbf9d4bda55158857 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 907e7632d20342bcbf9d4bda55158857, State: Initialized, Role: FOLLOWER
I20260620 18:42:07.165458 9651 consensus_queue.cc:260] T 00000000000000000000000000000000 P 907e7632d20342bcbf9d4bda55158857 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907e7632d20342bcbf9d4bda55158857" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 40021 } }
I20260620 18:42:07.165773 9651 raft_consensus.cc:388] T 00000000000000000000000000000000 P 907e7632d20342bcbf9d4bda55158857 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:42:07.165940 9651 raft_consensus.cc:482] T 00000000000000000000000000000000 P 907e7632d20342bcbf9d4bda55158857 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:42:07.166139 9651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 907e7632d20342bcbf9d4bda55158857 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:07.168437 9651 raft_consensus.cc:504] T 00000000000000000000000000000000 P 907e7632d20342bcbf9d4bda55158857 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907e7632d20342bcbf9d4bda55158857" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 40021 } }
I20260620 18:42:07.168939 9651 leader_election.cc:302] T 00000000000000000000000000000000 P 907e7632d20342bcbf9d4bda55158857 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 907e7632d20342bcbf9d4bda55158857; no voters:
I20260620 18:42:07.169500 9651 leader_election.cc:288] T 00000000000000000000000000000000 P 907e7632d20342bcbf9d4bda55158857 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:42:07.169652 9656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 907e7632d20342bcbf9d4bda55158857 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:42:07.170826 9656 raft_consensus.cc:686] T 00000000000000000000000000000000 P 907e7632d20342bcbf9d4bda55158857 [term 1 LEADER]: Becoming Leader. State: Replica: 907e7632d20342bcbf9d4bda55158857, State: Running, Role: LEADER
I20260620 18:42:07.171381 9656 consensus_queue.cc:237] T 00000000000000000000000000000000 P 907e7632d20342bcbf9d4bda55158857 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907e7632d20342bcbf9d4bda55158857" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 40021 } }
I20260620 18:42:07.171768 9651 sys_catalog.cc:564] T 00000000000000000000000000000000 P 907e7632d20342bcbf9d4bda55158857 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:42:07.174091 9658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 907e7632d20342bcbf9d4bda55158857 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 907e7632d20342bcbf9d4bda55158857. Latest consensus state: current_term: 1 leader_uuid: "907e7632d20342bcbf9d4bda55158857" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907e7632d20342bcbf9d4bda55158857" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 40021 } } }
I20260620 18:42:07.174504 9658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 907e7632d20342bcbf9d4bda55158857 [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:07.175031 9657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 907e7632d20342bcbf9d4bda55158857 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "907e7632d20342bcbf9d4bda55158857" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907e7632d20342bcbf9d4bda55158857" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 40021 } } }
I20260620 18:42:07.175343 9657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 907e7632d20342bcbf9d4bda55158857 [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:07.177695 9664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:42:07.182723 9664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:42:07.191548 9664 catalog_manager.cc:1347] Generated new cluster ID: 9e0a0a9e34b64f73aca676fa77094eb8
I20260620 18:42:07.191681 9664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:42:07.227865 9664 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:42:07.229243 9664 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:42:07.238467 9664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 907e7632d20342bcbf9d4bda55158857: Generated new TSK 0
I20260620 18:42:07.239284 9664 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab.
Entry for principal kudu/127.5.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab.
I20260620 18:42:07.297996 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.129:0
--local_ip_for_outbound_sockets=127.5.225.129
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:40021
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:36153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:07.628139 9677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:07.628461 9677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:07.628645 9677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:07.635900 9677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:07.636045 9677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:07.636234 9677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.129
I20260620 18:42:07.645596 9677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:36153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:40021
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.129
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:07.646867 9677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:07.648141 9677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:07.660701 9685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:07.660789 9683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:07.660821 9682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:07.661945 9677 server_base.cc:1034] running on GCE node
I20260620 18:42:07.662648 9677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:07.663869 9677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:07.665102 9677 hybrid_clock.cc:648] HybridClock initialized: now 1781980927665057 us; error 55 us; skew 500 ppm
I20260620 18:42:07.665580 9677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:07 dist-test-slave-k6lg krb5kdc[9571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980927, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:07.668766 9677 init.cc:377] Logged in from keytab as kudu/127.5.225.129@KRBTEST.COM (short username kudu)
I20260620 18:42:07.670156 9677 webserver.cc:460] Webserver started at http://127.5.225.129:34383/ using document root <none> and password file <none>
I20260620 18:42:07.670887 9677 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:07.671030 9677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:07.671340 9677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:07.674250 9677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-0/data/instance:
uuid: "1601658aa90a4b14a17df3cac3ea340d"
format_stamp: "Formatted at 2026-06-20 18:42:07 on dist-test-slave-k6lg"
server_key: "9e95f34cca64e01d632adba827c38c04"
server_key_iv: "2af24410011460cb385abe3c7975818f"
server_key_version: "encryptionkey@0"
I20260620 18:42:07.675017 9677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance:
uuid: "1601658aa90a4b14a17df3cac3ea340d"
format_stamp: "Formatted at 2026-06-20 18:42:07 on dist-test-slave-k6lg"
server_key: "9e95f34cca64e01d632adba827c38c04"
server_key_iv: "2af24410011460cb385abe3c7975818f"
server_key_version: "encryptionkey@0"
I20260620 18:42:07.680081 9677 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:42:07.683319 9692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:07.684583 9677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:42:07.684756 9677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-0/wal
uuid: "1601658aa90a4b14a17df3cac3ea340d"
format_stamp: "Formatted at 2026-06-20 18:42:07 on dist-test-slave-k6lg"
server_key: "9e95f34cca64e01d632adba827c38c04"
server_key_iv: "2af24410011460cb385abe3c7975818f"
server_key_version: "encryptionkey@0"
I20260620 18:42:07.684999 9677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:07.699652 9677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:07.703541 9677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:07.703840 9677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:07.704766 9677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:07.706048 9677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:07.706125 9677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:07.706321 9677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:07.706390 9677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:07.736651 9677 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.129:41937
I20260620 18:42:07.736696 9804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.129:41937 every 8 connection(s)
I20260620 18:42:07.738108 9677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
I20260620 18:42:07.748235 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 9677
I20260620 18:42:07.748508 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance
I20260620 18:42:07.749212 6022 external_mini_cluster.cc:1428] Setting key b4bfd966e04eca374900f1820de9a62e
Jun 20 18:42:07 dist-test-slave-k6lg krb5kdc[9571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980927, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:42:07.758785 9807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:07.740746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:39895 (local address 127.5.225.190:40021)
0620 18:42:07.741190 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:07.741196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:07.742156 (+ 960us) server_negotiation.cc:407] Connection header received
0620 18:42:07.743900 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:07.743904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:07.743968 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:07.744064 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:07.745443 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:07.746135 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:07.747639 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:07.748221 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:07.748353 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:07.752029 (+ 3676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:07.752039 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:07.752046 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:07.752091 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:07.755653 (+ 3562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:07.756384 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:07.756393 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:07.756399 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:07.756552 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:07.756961 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:07.756969 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:07.756974 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:07.757278 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:07.757391 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:07.758345 (+ 954us) server_negotiation.cc:299] Negotiation successful
0620 18:42:07.758521 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":197,"threads_started":1}
I20260620 18:42:07.760331 9808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:07.741324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:40021 (local address 127.5.225.129:39895)
0620 18:42:07.741993 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:07.742036 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:07.743663 (+ 1627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:07.744270 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:07.744278 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:07.744754 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:07.745302 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:07.745320 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:07.746314 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:07.746318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:07.747525 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:07.747536 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:07.748475 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:07.748481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:07.748726 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:07.749415 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:07.749447 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:07.751813 (+ 2366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:07.755810 (+ 3997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:07.755816 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:07.755830 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:07.756262 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:07.756664 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:07.756667 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:07.756669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:07.756849 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:07.757410 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:07.757420 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:07.757910 (+ 490us) client_negotiation.cc:769] Sending connection context
0620 18:42:07.758264 (+ 354us) client_negotiation.cc:240] Negotiation successful
0620 18:42:07.758860 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":259,"threads_started":1}
I20260620 18:42:07.762023 9805 heartbeater.cc:347] Connected to a master server at 127.5.225.190:40021
I20260620 18:42:07.762385 9805 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:07.763252 9805 heartbeater.cc:511] Master 127.5.225.190:40021 requested a full tablet report, sending...
I20260620 18:42:07.765550 9616 ts_manager.cc:194] Registered new tserver with Master: 1601658aa90a4b14a17df3cac3ea340d (127.5.225.129:41937)
I20260620 18:42:07.767030 9616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.129@KRBTEST.COM'} at 127.5.225.129:39895
WARNING: no policy specified for kudu/127.5.225.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab.
Entry for principal kudu/127.5.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab.
I20260620 18:42:07.795083 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.130:0
--local_ip_for_outbound_sockets=127.5.225.130
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:40021
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:36153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:08.117460 9811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:08.117836 9811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:08.118012 9811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:08.125383 9811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:08.125547 9811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:08.125743 9811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.130
I20260620 18:42:08.135744 9811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:36153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:40021
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.130
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:08.137009 9811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:08.138324 9811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:08.151170 9817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:08.151124 9816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:08.151213 9819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:08.152112 9811 server_base.cc:1034] running on GCE node
I20260620 18:42:08.152724 9811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:08.153657 9811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:08.154884 9811 hybrid_clock.cc:648] HybridClock initialized: now 1781980928154824 us; error 55 us; skew 500 ppm
I20260620 18:42:08.155200 9811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:08 dist-test-slave-k6lg krb5kdc[9571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980928, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:08.158340 9811 init.cc:377] Logged in from keytab as kudu/127.5.225.130@KRBTEST.COM (short username kudu)
I20260620 18:42:08.159739 9811 webserver.cc:460] Webserver started at http://127.5.225.130:35259/ using document root <none> and password file <none>
I20260620 18:42:08.160362 9811 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:08.160460 9811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:08.160707 9811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:08.163537 9811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-1/data/instance:
uuid: "3ddc2d63e1e54d4f90b2691c01c6efe2"
format_stamp: "Formatted at 2026-06-20 18:42:08 on dist-test-slave-k6lg"
server_key: "a239054a8f503b6ab5d9bad6c2886c3f"
server_key_iv: "902147f02815ac60c7aafb1963bbf7bf"
server_key_version: "encryptionkey@0"
I20260620 18:42:08.164271 9811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance:
uuid: "3ddc2d63e1e54d4f90b2691c01c6efe2"
format_stamp: "Formatted at 2026-06-20 18:42:08 on dist-test-slave-k6lg"
server_key: "a239054a8f503b6ab5d9bad6c2886c3f"
server_key_iv: "902147f02815ac60c7aafb1963bbf7bf"
server_key_version: "encryptionkey@0"
I20260620 18:42:08.169047 9811 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 18:42:08.172601 9826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:08.173945 9811 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 18:42:08.174113 9811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-1/wal
uuid: "3ddc2d63e1e54d4f90b2691c01c6efe2"
format_stamp: "Formatted at 2026-06-20 18:42:08 on dist-test-slave-k6lg"
server_key: "a239054a8f503b6ab5d9bad6c2886c3f"
server_key_iv: "902147f02815ac60c7aafb1963bbf7bf"
server_key_version: "encryptionkey@0"
I20260620 18:42:08.174372 9811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:08.189301 9811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:08.193213 9811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:08.193487 9811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:08.194517 9811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:08.195791 9811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:08.195891 9811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:08.196041 9811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:08.196132 9811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:08.230343 9811 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.130:46049
I20260620 18:42:08.230361 9938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.130:46049 every 8 connection(s)
I20260620 18:42:08.231658 9811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
I20260620 18:42:08.233547 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 9811
I20260620 18:42:08.233769 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance
I20260620 18:42:08.234340 6022 external_mini_cluster.cc:1428] Setting key 88132f60a57a11409ff390fce8a24615
Jun 20 18:42:08 dist-test-slave-k6lg krb5kdc[9571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980928, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:42:08.254271 9807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:08.234345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:38607 (local address 127.5.225.190:40021)
0620 18:42:08.234556 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:08.234563 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:08.236033 (+ 1470us) server_negotiation.cc:407] Connection header received
0620 18:42:08.237976 (+ 1943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:08.237984 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:08.238075 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:08.238216 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:08.239692 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:08.240835 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:08.242461 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:08.242905 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:08.243052 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:08.247155 (+ 4103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:08.247164 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:08.247172 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:08.247222 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:08.250838 (+ 3616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:08.251769 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:08.251778 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:08.251783 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:08.251949 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:08.252392 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:08.252399 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:08.252405 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:08.252788 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:08.252954 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:08.253613 (+ 659us) server_negotiation.cc:299] Negotiation successful
0620 18:42:08.253852 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
WARNING: no policy specified for kudu/127.5.225.131@KRBTEST.COM; defaulting to no policy
I20260620 18:42:08.254994 9941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:08.234986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:40021 (local address 127.5.225.130:38607)
0620 18:42:08.235851 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:08.235907 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:08.237697 (+ 1790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:08.238438 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:08.238446 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:08.238940 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:08.239489 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:08.239506 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:08.240989 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:08.240992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:08.242316 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:08.242328 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:08.243147 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:08.243154 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:08.243479 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:08.244188 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:08.244221 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:08.246879 (+ 2658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:08.250981 (+ 4102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:08.250988 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:08.251001 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:08.251622 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:08.252073 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:08.252076 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:08.252078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:08.252267 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:08.252909 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:08.252915 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:08.253230 (+ 315us) client_negotiation.cc:769] Sending connection context
0620 18:42:08.253443 (+ 213us) client_negotiation.cc:240] Negotiation successful
0620 18:42:08.253816 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":327,"threads_started":1}
I20260620 18:42:08.256508 9939 heartbeater.cc:347] Connected to a master server at 127.5.225.190:40021
I20260620 18:42:08.256824 9939 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.131@KRBTEST.COM" created.
I20260620 18:42:08.257578 9939 heartbeater.cc:511] Master 127.5.225.190:40021 requested a full tablet report, sending...
I20260620 18:42:08.259096 9616 ts_manager.cc:194] Registered new tserver with Master: 3ddc2d63e1e54d4f90b2691c01c6efe2 (127.5.225.130:46049)
I20260620 18:42:08.259904 9616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.130@KRBTEST.COM'} at 127.5.225.130:38607
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab.
Entry for principal kudu/127.5.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab.
I20260620 18:42:08.281220 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.131:0
--local_ip_for_outbound_sockets=127.5.225.131
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:40021
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:36153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:08.609134 9944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:08.609411 9944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:08.609524 9944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:08.616907 9944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:08.617017 9944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:08.617131 9944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.131
I20260620 18:42:08.626924 9944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:36153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:40021
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.131
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:08.628304 9944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:08.629614 9944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:08.642524 9949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:08.642524 9952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:08.642532 9950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:08.643841 9944 server_base.cc:1034] running on GCE node
I20260620 18:42:08.644454 9944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:08.645419 9944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:08.646652 9944 hybrid_clock.cc:648] HybridClock initialized: now 1781980928646584 us; error 60 us; skew 500 ppm
I20260620 18:42:08.646977 9944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:08 dist-test-slave-k6lg krb5kdc[9571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980928, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:08.650377 9944 init.cc:377] Logged in from keytab as kudu/127.5.225.131@KRBTEST.COM (short username kudu)
I20260620 18:42:08.651934 9944 webserver.cc:460] Webserver started at http://127.5.225.131:43853/ using document root <none> and password file <none>
I20260620 18:42:08.652634 9944 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:08.652783 9944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:08.653097 9944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:08.655997 9944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-2/data/instance:
uuid: "4f54aeeb19614ec685aa405122af66ac"
format_stamp: "Formatted at 2026-06-20 18:42:08 on dist-test-slave-k6lg"
server_key: "d769d8cd93526712a9ad9e2581471905"
server_key_iv: "4008461fab41d59b155d0fee81b009a0"
server_key_version: "encryptionkey@0"
I20260620 18:42:08.656796 9944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance:
uuid: "4f54aeeb19614ec685aa405122af66ac"
format_stamp: "Formatted at 2026-06-20 18:42:08 on dist-test-slave-k6lg"
server_key: "d769d8cd93526712a9ad9e2581471905"
server_key_iv: "4008461fab41d59b155d0fee81b009a0"
server_key_version: "encryptionkey@0"
I20260620 18:42:08.661718 9944 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:42:08.665091 9959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:08.666440 9944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:42:08.666608 9944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-2/wal
uuid: "4f54aeeb19614ec685aa405122af66ac"
format_stamp: "Formatted at 2026-06-20 18:42:08 on dist-test-slave-k6lg"
server_key: "d769d8cd93526712a9ad9e2581471905"
server_key_iv: "4008461fab41d59b155d0fee81b009a0"
server_key_version: "encryptionkey@0"
I20260620 18:42:08.666798 9944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:08.678512 9944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:08.682574 9944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:08.682799 9944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:08.683653 9944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:08.684942 9944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:08.685011 9944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:08.685093 9944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:08.685143 9944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:08.718124 9944 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.131:35871
I20260620 18:42:08.718189 10071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.131:35871 every 8 connection(s)
I20260620 18:42:08.719420 9944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
I20260620 18:42:08.729462 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 9944
I20260620 18:42:08.729651 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance
I20260620 18:42:08.730073 6022 external_mini_cluster.cc:1428] Setting key fd43f2e7b9784d388387b40fab6d332f
Jun 20 18:42:08 dist-test-slave-k6lg krb5kdc[9571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980928, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:42:08.740294 9807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:08.722054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:58563 (local address 127.5.225.190:40021)
0620 18:42:08.722282 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:08.722287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:08.723445 (+ 1158us) server_negotiation.cc:407] Connection header received
0620 18:42:08.725258 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:08.725264 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:08.725354 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:08.725464 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:08.726989 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:08.727758 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:08.729395 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:08.730017 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:08.730135 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:08.734268 (+ 4133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:08.734273 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:08.734277 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:08.734312 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:08.736994 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:08.737877 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:08.737881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:08.737883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:08.738028 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:08.738522 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:08.738525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:08.738527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:08.738801 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:08.738971 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:08.739787 (+ 816us) server_negotiation.cc:299] Negotiation successful
0620 18:42:08.739951 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:42:08.741400 10074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:08.722628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:40021 (local address 127.5.225.131:58563)
0620 18:42:08.723286 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:08.723327 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:08.724983 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:08.725701 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:08.725709 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:08.726259 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:08.726823 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:08.726840 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:08.727913 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:08.727918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:08.729223 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:08.729245 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:08.730269 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:08.730278 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:08.730562 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:08.731376 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:08.731405 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:08.734022 (+ 2617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:08.737129 (+ 3107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:08.737137 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:08.737154 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:08.737747 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:08.738138 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:08.738143 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:08.738150 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:08.738384 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:08.738977 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:08.738986 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:08.739357 (+ 371us) client_negotiation.cc:769] Sending connection context
0620 18:42:08.739652 (+ 295us) client_negotiation.cc:240] Negotiation successful
0620 18:42:08.740155 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":255,"threads_started":1}
I20260620 18:42:08.742805 10072 heartbeater.cc:347] Connected to a master server at 127.5.225.190:40021
I20260620 18:42:08.743069 10072 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:08.743849 10072 heartbeater.cc:511] Master 127.5.225.190:40021 requested a full tablet report, sending...
I20260620 18:42:08.745397 9615 ts_manager.cc:194] Registered new tserver with Master: 4f54aeeb19614ec685aa405122af66ac (127.5.225.131:35871)
I20260620 18:42:08.746313 9615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.131@KRBTEST.COM'} at 127.5.225.131:58563
I20260620 18:42:08.757767 6022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:42:08.770589 9805 heartbeater.cc:503] Master 127.5.225.190:40021 was elected leader, sending a full tablet report...
Jun 20 18:42:08 dist-test-slave-k6lg krb5kdc[9571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 18:42:08 dist-test-slave-k6lg krb5kdc[9571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.129@KRBTEST.COM
I20260620 18:42:08.805362 10079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:08.790670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44072 (local address 127.5.225.129:41937)
0620 18:42:08.791316 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:08.791324 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:08.791353 (+ 29us) server_negotiation.cc:407] Connection header received
0620 18:42:08.791491 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:08.791499 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:08.791729 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:08.791861 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:08.793159 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:08.794242 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:08.795457 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:08.796021 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:08.796127 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:08.799838 (+ 3711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:08.799849 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:08.799863 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:08.799901 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:08.802913 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:08.803521 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:08.803527 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:08.803530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:08.803670 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:08.803967 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:08.803971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:08.803973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:08.804301 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:08.804509 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:08.804777 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 18:42:08.804959 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":313,"threads_started":1}
Jun 20 18:42:08 dist-test-slave-k6lg krb5kdc[9571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:42:08.821910 9807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:08.809367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50366 (local address 127.5.225.190:40021)
0620 18:42:08.809575 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:08.809579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:08.809687 (+ 108us) server_negotiation.cc:407] Connection header received
0620 18:42:08.809820 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:08.809824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:08.809884 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:08.809963 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:08.811229 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:08.811948 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:08.813166 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:08.813554 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:08.813641 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:08.817276 (+ 3635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:08.817280 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:08.817283 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:08.817315 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:08.819694 (+ 2379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:08.820275 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:08.820279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:08.820281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:08.820399 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:08.820723 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:08.820726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:08.820728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:08.821004 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:08.821187 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:08.821427 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 18:42:08.821575 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260620 18:42:08.822650 9615 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50366
I20260620 18:42:08.823731 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 9677
I20260620 18:42:08.835741 10079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:08.824616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44086 (local address 127.5.225.129:41937)
0620 18:42:08.824780 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:08.824784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:08.824966 (+ 182us) server_negotiation.cc:407] Connection header received
0620 18:42:08.825098 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:08.825102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:08.825160 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:08.825239 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:08.826454 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:08.827128 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:08.828280 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:08.828674 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:08.828761 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:08.830968 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:08.830972 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:08.830976 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:08.831004 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:08.833357 (+ 2353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:08.833928 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:08.833931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:08.833933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:08.834042 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:08.834444 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:08.834448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:08.834450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:08.834733 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:08.834923 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:08.835326 (+ 403us) server_negotiation.cc:299] Negotiation successful
0620 18:42:08.835499 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 18:42:08.836498 9800 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:08.919260 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 9677
I20260620 18:42:08.935855 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 9811
Jun 20 18:42:08 dist-test-slave-k6lg krb5kdc[9571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.130@KRBTEST.COM
I20260620 18:42:08.951555 10084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:08.936788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51844 (local address 127.5.225.130:46049)
0620 18:42:08.937385 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:08.937392 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:08.937417 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:42:08.937478 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:08.937484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:08.937694 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:08.937828 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:08.939065 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:08.940104 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:08.941335 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:08.941897 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:08.941992 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:08.946632 (+ 4640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:08.946644 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:08.946657 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:08.946689 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:08.949251 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:08.949850 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:08.949857 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:08.949859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:08.949969 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:08.950336 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:08.950339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:08.950341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:08.950643 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:08.950864 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:08.951086 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 18:42:08.951269 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":312,"threads_started":1}
I20260620 18:42:08.952780 9934 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:09.035075 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 9811
I20260620 18:42:09.051461 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 9944
Jun 20 18:42:09 dist-test-slave-k6lg krb5kdc[9571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.131@KRBTEST.COM
I20260620 18:42:09.067905 10086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:09.052568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50484 (local address 127.5.225.131:35871)
0620 18:42:09.053194 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:09.053204 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:09.053244 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:42:09.053326 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:09.053334 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:09.053616 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:09.053761 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:09.055067 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:09.056059 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:09.057330 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:09.057880 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:09.057975 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:09.062298 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:09.062309 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:09.062323 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:09.062359 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:09.065158 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:09.065903 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:09.065911 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:09.065913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:09.066041 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:09.066470 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:09.066474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:09.066476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:09.066796 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:09.067025 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:09.067297 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 18:42:09.067532 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":276,"threads_started":1}
I20260620 18:42:09.069175 10067 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:09.151598 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 9944
I20260620 18:42:09.167359 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 9584
I20260620 18:42:09.168519 9646 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:09.271988 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 9584
2026-06-20T18:42:09Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3848 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:42:09 dist-test-slave-k6lg krb5kdc[10091](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:42:09 dist-test-slave-k6lg krb5kdc[10091](info): set up 2 sockets
Jun 20 18:42:09 dist-test-slave-k6lg krb5kdc[10091](info): commencing operation
krb5kdc: starting...
W20260620 18:42:10.390661 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.005s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:42:10 dist-test-slave-k6lg krb5kdc[10091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:42:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:42:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
I20260620 18:42:10.585675 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:32883
--webserver_interface=127.5.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:35243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:32883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:10.899411 10104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:10.899698 10104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:10.899883 10104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:10.907061 10104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:42:10.907174 10104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:10.907284 10104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:10.907388 10104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:42:10.907477 10104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:42:10.918742 10104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:35243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:32883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:32883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:10.920122 10104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:10.921505 10104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:10.934252 10112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:10.934319 10109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:10.934341 10110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:10.935109 10104 server_base.cc:1034] running on GCE node
I20260620 18:42:10.935983 10104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:10.937367 10104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:10.938648 10104 hybrid_clock.cc:648] HybridClock initialized: now 1781980930938588 us; error 46 us; skew 500 ppm
I20260620 18:42:10.939009 10104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:10 dist-test-slave-k6lg krb5kdc[10091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980930, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:10.942404 10104 init.cc:377] Logged in from keytab as kudu/127.5.225.190@KRBTEST.COM (short username kudu)
I20260620 18:42:10.943698 10104 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:10.943836 10104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:10.944171 10104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:10.947278 10104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "46465637b04e42999bd9bb839f8d37eb"
format_stamp: "Formatted at 2026-06-20 18:42:10 on dist-test-slave-k6lg"
server_key: "1c4f938706a406aa0251b1c0379f8b2c"
server_key_iv: "fab574174fd5cd6f55b6d97b1563f5c8"
server_key_version: "encryptionkey@0"
I20260620 18:42:10.948141 10104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "46465637b04e42999bd9bb839f8d37eb"
format_stamp: "Formatted at 2026-06-20 18:42:10 on dist-test-slave-k6lg"
server_key: "1c4f938706a406aa0251b1c0379f8b2c"
server_key_iv: "fab574174fd5cd6f55b6d97b1563f5c8"
server_key_version: "encryptionkey@0"
I20260620 18:42:10.953208 10104 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 18:42:10.956666 10118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:10.957962 10104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:42:10.958199 10104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "46465637b04e42999bd9bb839f8d37eb"
format_stamp: "Formatted at 2026-06-20 18:42:10 on dist-test-slave-k6lg"
server_key: "1c4f938706a406aa0251b1c0379f8b2c"
server_key_iv: "fab574174fd5cd6f55b6d97b1563f5c8"
server_key_version: "encryptionkey@0"
I20260620 18:42:10.958461 10104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:10.998529 10104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:11.002760 10104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:11.003067 10104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:11.021795 10104 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:32883
I20260620 18:42:11.021816 10169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:32883 every 8 connection(s)
I20260620 18:42:11.022763 10104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:42:11.023255 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 10104
I20260620 18:42:11.023491 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:42:11.024274 6022 external_mini_cluster.cc:1428] Setting key 3665b9ad2c8e2c80287b9bea1db5a106
I20260620 18:42:11.027212 10170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:11.037492 10170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46465637b04e42999bd9bb839f8d37eb: Bootstrap starting.
Jun 20 18:42:11 dist-test-slave-k6lg krb5kdc[10091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:42:11.040831 10170 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46465637b04e42999bd9bb839f8d37eb: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:11.042038 10170 log.cc:824] T 00000000000000000000000000000000 P 46465637b04e42999bd9bb839f8d37eb: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:11.044785 10170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46465637b04e42999bd9bb839f8d37eb: No bootstrap required, opened a new log
I20260620 18:42:11.046464 10173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:11.026360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47836 (local address 127.5.225.190:32883)
0620 18:42:11.027202 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:11.027215 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:11.027270 (+ 55us) server_negotiation.cc:407] Connection header received
0620 18:42:11.028946 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:11.028967 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:11.029363 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:11.029830 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:11.031342 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:11.032661 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:11.033924 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:11.034670 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:11.034879 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:11.038880 (+ 4001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:11.038891 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:11.038907 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:11.038947 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:11.042382 (+ 3435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:11.043094 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:11.043102 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:11.043111 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:11.043290 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:11.043681 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:11.043686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:11.043689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:11.044223 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:11.044541 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:11.044976 (+ 435us) server_negotiation.cc:299] Negotiation successful
0620 18:42:11.045406 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":405,"threads_started":1}
I20260620 18:42:11.053730 10170 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46465637b04e42999bd9bb839f8d37eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46465637b04e42999bd9bb839f8d37eb" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 32883 } }
I20260620 18:42:11.054129 10170 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46465637b04e42999bd9bb839f8d37eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:11.054299 10170 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46465637b04e42999bd9bb839f8d37eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46465637b04e42999bd9bb839f8d37eb, State: Initialized, Role: FOLLOWER
I20260620 18:42:11.054862 10170 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46465637b04e42999bd9bb839f8d37eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46465637b04e42999bd9bb839f8d37eb" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 32883 } }
I20260620 18:42:11.055094 10170 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46465637b04e42999bd9bb839f8d37eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:42:11.055248 10170 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46465637b04e42999bd9bb839f8d37eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:42:11.055429 10170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46465637b04e42999bd9bb839f8d37eb [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:11.057665 10170 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46465637b04e42999bd9bb839f8d37eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46465637b04e42999bd9bb839f8d37eb" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 32883 } }
I20260620 18:42:11.058204 10170 leader_election.cc:302] T 00000000000000000000000000000000 P 46465637b04e42999bd9bb839f8d37eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46465637b04e42999bd9bb839f8d37eb; no voters:
I20260620 18:42:11.058916 10170 leader_election.cc:288] T 00000000000000000000000000000000 P 46465637b04e42999bd9bb839f8d37eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:42:11.059072 10175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46465637b04e42999bd9bb839f8d37eb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:42:11.059841 10175 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46465637b04e42999bd9bb839f8d37eb [term 1 LEADER]: Becoming Leader. State: Replica: 46465637b04e42999bd9bb839f8d37eb, State: Running, Role: LEADER
I20260620 18:42:11.060392 10175 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46465637b04e42999bd9bb839f8d37eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46465637b04e42999bd9bb839f8d37eb" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 32883 } }
I20260620 18:42:11.061018 10170 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46465637b04e42999bd9bb839f8d37eb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:42:11.063309 10176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46465637b04e42999bd9bb839f8d37eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46465637b04e42999bd9bb839f8d37eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46465637b04e42999bd9bb839f8d37eb" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 32883 } } }
I20260620 18:42:11.063282 10177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46465637b04e42999bd9bb839f8d37eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46465637b04e42999bd9bb839f8d37eb. Latest consensus state: current_term: 1 leader_uuid: "46465637b04e42999bd9bb839f8d37eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46465637b04e42999bd9bb839f8d37eb" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 32883 } } }
I20260620 18:42:11.063709 10176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46465637b04e42999bd9bb839f8d37eb [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:11.063705 10177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46465637b04e42999bd9bb839f8d37eb [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:11.065393 10180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:42:11.071017 10180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:42:11.080601 10180 catalog_manager.cc:1347] Generated new cluster ID: 929df10e2dd944c3a4be96ceafc91e75
I20260620 18:42:11.080739 10180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:42:11.098136 10180 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:42:11.100021 10180 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:42:11.108528 10180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46465637b04e42999bd9bb839f8d37eb: Generated new TSK 0
I20260620 18:42:11.109572 10180 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab.
Entry for principal kudu/127.5.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab.
I20260620 18:42:11.166527 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.129:0
--local_ip_for_outbound_sockets=127.5.225.129
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:32883
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:35243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:11.499927 10196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:11.500231 10196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:11.500408 10196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:11.507616 10196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:11.507748 10196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:11.507930 10196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.129
I20260620 18:42:11.517515 10196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:35243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:32883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.129
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:11.518865 10196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:11.520213 10196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:11.533377 10201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:11.533305 10204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:11.533326 10202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:11.534163 10196 server_base.cc:1034] running on GCE node
I20260620 18:42:11.534865 10196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:11.535888 10196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:11.537149 10196 hybrid_clock.cc:648] HybridClock initialized: now 1781980931537116 us; error 33 us; skew 500 ppm
I20260620 18:42:11.537490 10196 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:11 dist-test-slave-k6lg krb5kdc[10091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980931, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:11.540791 10196 init.cc:377] Logged in from keytab as kudu/127.5.225.129@KRBTEST.COM (short username kudu)
I20260620 18:42:11.541929 10196 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:11.542130 10196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:11.542517 10196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:11.545567 10196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-0/data/instance:
uuid: "ca3fc488ac5049adb4badd99378bc814"
format_stamp: "Formatted at 2026-06-20 18:42:11 on dist-test-slave-k6lg"
server_key: "b620c550d97e71ed081f78a424aa80d0"
server_key_iv: "b8fbeaaf3a24ebc4701cea162a4347a7"
server_key_version: "encryptionkey@0"
I20260620 18:42:11.546474 10196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance:
uuid: "ca3fc488ac5049adb4badd99378bc814"
format_stamp: "Formatted at 2026-06-20 18:42:11 on dist-test-slave-k6lg"
server_key: "b620c550d97e71ed081f78a424aa80d0"
server_key_iv: "b8fbeaaf3a24ebc4701cea162a4347a7"
server_key_version: "encryptionkey@0"
I20260620 18:42:11.551518 10196 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:42:11.554939 10210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:11.556316 10196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:42:11.556486 10196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-0/wal
uuid: "ca3fc488ac5049adb4badd99378bc814"
format_stamp: "Formatted at 2026-06-20 18:42:11 on dist-test-slave-k6lg"
server_key: "b620c550d97e71ed081f78a424aa80d0"
server_key_iv: "b8fbeaaf3a24ebc4701cea162a4347a7"
server_key_version: "encryptionkey@0"
I20260620 18:42:11.556716 10196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:11.615864 10196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:11.620143 10196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:11.620427 10196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:11.621379 10196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:11.622749 10196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:11.622838 10196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:11.622989 10196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:11.623083 10196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:11.655874 10196 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.129:45199
I20260620 18:42:11.655925 10322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.129:45199 every 8 connection(s)
I20260620 18:42:11.656795 10196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
I20260620 18:42:11.665895 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 10196
I20260620 18:42:11.666126 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance
I20260620 18:42:11.666775 6022 external_mini_cluster.cc:1428] Setting key 9c0aef7af3545bc72235528e0e80aafa
Jun 20 18:42:11 dist-test-slave-k6lg krb5kdc[10091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980931, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:42:11.677361 10325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:11.659397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:33555 (local address 127.5.225.190:32883)
0620 18:42:11.659889 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:11.659894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:11.661060 (+ 1166us) server_negotiation.cc:407] Connection header received
0620 18:42:11.663127 (+ 2067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:11.663133 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:11.663226 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:11.663363 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:11.664829 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:11.665530 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:11.667290 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:11.667731 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:11.667851 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:11.671832 (+ 3981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:11.671841 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:11.671846 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:11.671890 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:11.674443 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:11.675145 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:11.675152 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:11.675155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:11.675278 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:11.675702 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:11.675713 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:11.675715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:11.675964 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:11.676118 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:11.676797 (+ 679us) server_negotiation.cc:299] Negotiation successful
0620 18:42:11.677005 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":229,"threads_started":1}
I20260620 18:42:11.678351 10326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:11.660065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:32883 (local address 127.5.225.129:33555)
0620 18:42:11.660867 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:11.660934 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:11.662864 (+ 1930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:11.663593 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:11.663602 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:11.664098 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:11.664659 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:11.664676 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:11.665698 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:11.665708 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:11.667132 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:11.667144 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:11.667948 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:11.667959 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:11.668157 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:11.668899 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:11.668934 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:11.671575 (+ 2641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:11.674582 (+ 3007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:11.674591 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:11.674607 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:11.675017 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:11.675397 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:11.675401 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:11.675406 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:11.675578 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:11.676095 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:11.676102 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:11.676429 (+ 327us) client_negotiation.cc:769] Sending connection context
0620 18:42:11.676679 (+ 250us) client_negotiation.cc:240] Negotiation successful
0620 18:42:11.677186 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":303,"threads_started":1}
I20260620 18:42:11.679953 10323 heartbeater.cc:347] Connected to a master server at 127.5.225.190:32883
I20260620 18:42:11.680243 10323 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:11.680877 10323 heartbeater.cc:511] Master 127.5.225.190:32883 requested a full tablet report, sending...
I20260620 18:42:11.682932 10135 ts_manager.cc:194] Registered new tserver with Master: ca3fc488ac5049adb4badd99378bc814 (127.5.225.129:45199)
I20260620 18:42:11.684326 10135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.129@KRBTEST.COM'} at 127.5.225.129:33555
WARNING: no policy specified for kudu/127.5.225.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab.
Entry for principal kudu/127.5.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab.
I20260620 18:42:11.709689 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.130:0
--local_ip_for_outbound_sockets=127.5.225.130
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:32883
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:35243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:12.035871 10329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:12.036218 10329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:12.036326 10329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:12.043857 10329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:12.043959 10329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:12.044073 10329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.130
I20260620 18:42:12.053785 10329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:35243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:32883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.130
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:12.055164 10329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:12.056493 10329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:12.069908 10337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:12.069988 10335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:12.070006 10334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:12.070801 10329 server_base.cc:1034] running on GCE node
I20260620 18:42:12.071446 10329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:12.072449 10329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:12.073669 10329 hybrid_clock.cc:648] HybridClock initialized: now 1781980932073616 us; error 40 us; skew 500 ppm
I20260620 18:42:12.073951 10329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:12 dist-test-slave-k6lg krb5kdc[10091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980932, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:12.079864 10329 init.cc:377] Logged in from keytab as kudu/127.5.225.130@KRBTEST.COM (short username kudu)
I20260620 18:42:12.080989 10329 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:12.081080 10329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:12.081344 10329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:12.084344 10329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-1/data/instance:
uuid: "02919670e5434459b173a97c8a0ba38e"
format_stamp: "Formatted at 2026-06-20 18:42:12 on dist-test-slave-k6lg"
server_key: "baa12bd51bd938a1a3dd114496f34315"
server_key_iv: "3778ec9c05c6a95221af36a50e32df25"
server_key_version: "encryptionkey@0"
I20260620 18:42:12.085106 10329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance:
uuid: "02919670e5434459b173a97c8a0ba38e"
format_stamp: "Formatted at 2026-06-20 18:42:12 on dist-test-slave-k6lg"
server_key: "baa12bd51bd938a1a3dd114496f34315"
server_key_iv: "3778ec9c05c6a95221af36a50e32df25"
server_key_version: "encryptionkey@0"
I20260620 18:42:12.090056 10329 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 18:42:12.093513 10343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:12.094889 10329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:42:12.095062 10329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-1/wal
uuid: "02919670e5434459b173a97c8a0ba38e"
format_stamp: "Formatted at 2026-06-20 18:42:12 on dist-test-slave-k6lg"
server_key: "baa12bd51bd938a1a3dd114496f34315"
server_key_iv: "3778ec9c05c6a95221af36a50e32df25"
server_key_version: "encryptionkey@0"
I20260620 18:42:12.095281 10329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:12.131246 10329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:12.135576 10329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:12.135852 10329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:12.136828 10329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:12.138242 10329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:12.138329 10329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:12.138451 10329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:12.138547 10329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:12.172143 10329 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.130:45077
I20260620 18:42:12.172196 10455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.130:45077 every 8 connection(s)
I20260620 18:42:12.173004 10329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
I20260620 18:42:12.178704 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 10329
I20260620 18:42:12.178908 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance
I20260620 18:42:12.179427 6022 external_mini_cluster.cc:1428] Setting key 908b01ff31f3128b89f73b6ebcd9693f
Jun 20 18:42:12 dist-test-slave-k6lg krb5kdc[10091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980932, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:42:12.194521 10325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:12.175369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:54847 (local address 127.5.225.190:32883)
0620 18:42:12.175579 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:12.175584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:12.176631 (+ 1047us) server_negotiation.cc:407] Connection header received
0620 18:42:12.178477 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:12.178481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:12.178552 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:12.178645 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:12.180602 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:12.181637 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:12.184164 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:12.184576 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:12.184676 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:12.188843 (+ 4167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:12.188847 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:12.188851 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:12.188883 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:12.191520 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:12.192277 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:12.192286 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:12.192289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:12.192406 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:12.192821 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:12.192826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:12.192829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:12.193088 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:12.193240 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:12.193911 (+ 671us) server_negotiation.cc:299] Negotiation successful
0620 18:42:12.194147 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 18:42:12.195663 10458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:12.175783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:32883 (local address 127.5.225.130:54847)
0620 18:42:12.176468 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:12.176506 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:12.178137 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:12.178915 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:12.178928 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:12.179644 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:12.180393 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:12.180418 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:12.181798 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:12.181804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:12.183998 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:12.184016 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:12.184796 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:12.184806 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:12.185135 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:12.185863 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:12.185896 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:12.188609 (+ 2713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:12.191683 (+ 3074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:12.191692 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:12.191708 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:12.192150 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:12.192513 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:12.192518 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:12.192522 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:12.192703 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:12.193223 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:12.193231 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:12.193545 (+ 314us) client_negotiation.cc:769] Sending connection context
0620 18:42:12.193820 (+ 275us) client_negotiation.cc:240] Negotiation successful
0620 18:42:12.194366 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":277,"threads_started":1}
WARNING: no policy specified for kudu/127.5.225.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.131@KRBTEST.COM" created.
I20260620 18:42:12.203806 10456 heartbeater.cc:347] Connected to a master server at 127.5.225.190:32883
I20260620 18:42:12.204166 10456 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:12.204864 10456 heartbeater.cc:511] Master 127.5.225.190:32883 requested a full tablet report, sending...
I20260620 18:42:12.206432 10135 ts_manager.cc:194] Registered new tserver with Master: 02919670e5434459b173a97c8a0ba38e (127.5.225.130:45077)
I20260620 18:42:12.207242 10135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.130@KRBTEST.COM'} at 127.5.225.130:54847
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab.
Entry for principal kudu/127.5.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab.
I20260620 18:42:12.229236 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.131:0
--local_ip_for_outbound_sockets=127.5.225.131
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:32883
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:35243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:12.544919 10461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:12.545192 10461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:12.545347 10461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:12.552470 10461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:12.552577 10461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:12.552763 10461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.131
I20260620 18:42:12.562772 10461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:35243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:32883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.131
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:12.564114 10461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:12.565397 10461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:12.578322 10467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:12.578348 10469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:12.578913 10466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:12.579670 10461 server_base.cc:1034] running on GCE node
I20260620 18:42:12.580241 10461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:12.581146 10461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:12.582365 10461 hybrid_clock.cc:648] HybridClock initialized: now 1781980932582326 us; error 52 us; skew 500 ppm
I20260620 18:42:12.582690 10461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:12 dist-test-slave-k6lg krb5kdc[10091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980932, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:12.585664 10461 init.cc:377] Logged in from keytab as kudu/127.5.225.131@KRBTEST.COM (short username kudu)
I20260620 18:42:12.586851 10461 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:12.586982 10461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:12.587294 10461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:12.590155 10461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-2/data/instance:
uuid: "cd94b43daf8042a08b5b91527f42d507"
format_stamp: "Formatted at 2026-06-20 18:42:12 on dist-test-slave-k6lg"
server_key: "793530461f50a652067d074f6b188c44"
server_key_iv: "09ce2aa0ff352de7d87aa7041fcf0b6e"
server_key_version: "encryptionkey@0"
I20260620 18:42:12.591022 10461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance:
uuid: "cd94b43daf8042a08b5b91527f42d507"
format_stamp: "Formatted at 2026-06-20 18:42:12 on dist-test-slave-k6lg"
server_key: "793530461f50a652067d074f6b188c44"
server_key_iv: "09ce2aa0ff352de7d87aa7041fcf0b6e"
server_key_version: "encryptionkey@0"
I20260620 18:42:12.595901 10461 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:42:12.599345 10475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:12.600837 10461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:42:12.601071 10461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-2/wal
uuid: "cd94b43daf8042a08b5b91527f42d507"
format_stamp: "Formatted at 2026-06-20 18:42:12 on dist-test-slave-k6lg"
server_key: "793530461f50a652067d074f6b188c44"
server_key_iv: "09ce2aa0ff352de7d87aa7041fcf0b6e"
server_key_version: "encryptionkey@0"
I20260620 18:42:12.601313 10461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:12.621995 10461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:12.626299 10461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:12.626585 10461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:12.627516 10461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:12.628882 10461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:12.629000 10461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:12.629129 10461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:12.629213 10461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:12.663386 10461 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.131:44991
I20260620 18:42:12.663398 10587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.131:44991 every 8 connection(s)
I20260620 18:42:12.664259 10461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
I20260620 18:42:12.667677 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 10461
I20260620 18:42:12.667914 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance
I20260620 18:42:12.668430 6022 external_mini_cluster.cc:1428] Setting key 531f1a6c357a8c782c572d654132a66e
Jun 20 18:42:12 dist-test-slave-k6lg krb5kdc[10091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980932, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:42:12.684412 10325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:12.666809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:44263 (local address 127.5.225.190:32883)
0620 18:42:12.667013 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:12.667020 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:12.668492 (+ 1472us) server_negotiation.cc:407] Connection header received
0620 18:42:12.670673 (+ 2181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:12.670678 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:12.670766 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:12.670872 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:12.672372 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:12.673125 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:12.674729 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:12.675153 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:12.675291 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:12.678641 (+ 3350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:12.678645 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:12.678648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:12.678680 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:12.681177 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:12.681885 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:12.681891 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:12.681894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:12.682052 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:12.682575 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:12.682582 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:12.682586 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:12.682914 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:12.683063 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:12.683983 (+ 920us) server_negotiation.cc:299] Negotiation successful
0620 18:42:12.684153 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:42:12.685374 10590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:12.667374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:32883 (local address 127.5.225.131:44263)
0620 18:42:12.668338 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:12.668374 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:12.670390 (+ 2016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:12.671100 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:12.671109 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:12.671663 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:12.672205 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:12.672221 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:12.673266 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:12.673273 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:12.674588 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:12.674600 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:12.675376 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:12.675385 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:12.675624 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:12.676231 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:12.676251 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:12.678453 (+ 2202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:12.681322 (+ 2869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:12.681329 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:12.681340 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:12.681769 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:12.682198 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:12.682205 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:12.682211 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:12.682429 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:12.683073 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:12.683082 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:12.683536 (+ 454us) client_negotiation.cc:769] Sending connection context
0620 18:42:12.683863 (+ 327us) client_negotiation.cc:240] Negotiation successful
0620 18:42:12.684244 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"thread_start_us":387,"threads_started":1}
I20260620 18:42:12.686761 10588 heartbeater.cc:347] Connected to a master server at 127.5.225.190:32883
I20260620 18:42:12.687029 10588 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:12.687116 10323 heartbeater.cc:503] Master 127.5.225.190:32883 was elected leader, sending a full tablet report...
I20260620 18:42:12.687736 10588 heartbeater.cc:511] Master 127.5.225.190:32883 requested a full tablet report, sending...
I20260620 18:42:12.689075 10135 ts_manager.cc:194] Registered new tserver with Master: cd94b43daf8042a08b5b91527f42d507 (127.5.225.131:44991)
I20260620 18:42:12.689885 10135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.131@KRBTEST.COM'} at 127.5.225.131:44263
I20260620 18:42:12.695914 6022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:42:12.713843 10325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:12.701079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47844 (local address 127.5.225.190:32883)
0620 18:42:12.701331 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:12.701340 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:12.701778 (+ 438us) server_negotiation.cc:407] Connection header received
0620 18:42:12.701913 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:12.701917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:12.701982 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:12.702060 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:12.703612 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:12.704313 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:12.705874 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:12.706368 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:12.706478 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:12.708795 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:12.708799 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:12.708803 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:12.708833 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:12.711418 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:12.712049 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:12.712052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:12.712055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:12.712173 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:12.712537 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:12.712540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:12.712542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:12.712884 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:12.713073 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:12.713387 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 18:42:12.713539 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 18:42:12.717911 10135 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47844:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:42:12.724694 10135 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:42:12.752046 10602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:12.737221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.130:45077 (local address 127.0.0.1:55726)
0620 18:42:12.738490 (+ 1269us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:12.738539 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:12.738767 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:12.740712 (+ 1945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:12.740720 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:12.740754 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:12.741457 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:12.741471 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:12.744717 (+ 3246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:12.744726 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:12.748381 (+ 3655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:12.748404 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:12.750855 (+ 2451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:12.750866 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:12.751122 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:12.751195 (+ 73us) client_negotiation.cc:769] Sending connection context
0620 18:42:12.751313 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 18:42:12.751552 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":922,"thread_start_us":1914,"threads_started":1}
I20260620 18:42:12.754014 10600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:12.737283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55726 (local address 127.5.225.130:45077)
0620 18:42:12.740110 (+ 2827us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:12.740119 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:12.740146 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:42:12.740243 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:12.740252 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:12.740466 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:12.740683 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:12.741743 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:12.744514 (+ 2771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:12.748683 (+ 4169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:12.750620 (+ 1937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:12.753164 (+ 2544us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:12.753385 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:12.753454 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 18:42:12.753622 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2677,"thread_start_us":2579,"threads_started":1}
I20260620 18:42:12.759276 10601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:12.737275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.131:44991 (local address 127.0.0.1:37382)
0620 18:42:12.739456 (+ 2181us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:12.739477 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:12.739595 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:12.751316 (+ 11721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:12.751322 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:12.751352 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:12.751776 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:12.751791 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:12.753937 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:12.753943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:12.755974 (+ 2031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:12.755989 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:12.758388 (+ 2399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:12.758396 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:12.758599 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:12.758637 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 18:42:12.758733 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 18:42:12.758879 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2034,"thread_start_us":1313,"threads_started":1}
I20260620 18:42:12.759140 10391 tablet_service.cc:1467] Processing CreateTablet for tablet ef990ff6a42c4c72a6e9b39cf26b7e6a (DEFAULT_TABLE table=test-table [id=15e08cef23864f85933a3bd641c302a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:42:12.760510 10603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:12.738236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37382 (local address 127.5.225.131:44991)
0620 18:42:12.750373 (+ 12137us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:12.750385 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:12.750422 (+ 37us) server_negotiation.cc:407] Connection header received
0620 18:42:12.750534 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:12.750543 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:12.750834 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:12.751042 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:12.752022 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:12.753766 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:12.756150 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:12.757794 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:12.757920 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:12.758228 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:12.759941 (+ 1713us) server_negotiation.cc:299] Negotiation successful
0620 18:42:12.760140 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11726,"thread_start_us":11917,"threads_started":1}
I20260620 18:42:12.761265 10391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef990ff6a42c4c72a6e9b39cf26b7e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:12.765093 10602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:12.741457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:45199 (local address 127.0.0.1:48936)
0620 18:42:12.752865 (+ 11408us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:12.752883 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:12.752996 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:12.757476 (+ 4480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:12.757496 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:12.757526 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:12.757773 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:12.757785 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:12.759909 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:12.759915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:12.762829 (+ 2914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:12.762842 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:12.764291 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:12.764297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:12.764487 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:12.764528 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 18:42:12.764625 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 18:42:12.764778 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11309,"thread_start_us":19838,"threads_started":1}
I20260620 18:42:12.764706 10523 tablet_service.cc:1467] Processing CreateTablet for tablet ef990ff6a42c4c72a6e9b39cf26b7e6a (DEFAULT_TABLE table=test-table [id=15e08cef23864f85933a3bd641c302a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:42:12.765384 10605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:12.741588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48936 (local address 127.5.225.129:45199)
0620 18:42:12.756611 (+ 15023us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:12.756620 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:12.756652 (+ 32us) server_negotiation.cc:407] Connection header received
0620 18:42:12.756760 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:12.756768 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:12.757021 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:12.757228 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:12.757963 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:12.759722 (+ 1759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:12.763044 (+ 3322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:12.764103 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:12.764279 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:12.764529 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:12.764824 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 18:42:12.764998 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14820,"thread_start_us":22969,"threads_started":1}
I20260620 18:42:12.766280 10523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef990ff6a42c4c72a6e9b39cf26b7e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:12.768985 10258 tablet_service.cc:1467] Processing CreateTablet for tablet ef990ff6a42c4c72a6e9b39cf26b7e6a (DEFAULT_TABLE table=test-table [id=15e08cef23864f85933a3bd641c302a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:42:12.770303 10258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef990ff6a42c4c72a6e9b39cf26b7e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:12.776706 10606 tablet_bootstrap.cc:492] T ef990ff6a42c4c72a6e9b39cf26b7e6a P 02919670e5434459b173a97c8a0ba38e: Bootstrap starting.
I20260620 18:42:12.780643 10606 tablet_bootstrap.cc:654] T ef990ff6a42c4c72a6e9b39cf26b7e6a P 02919670e5434459b173a97c8a0ba38e: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:12.781685 10607 tablet_bootstrap.cc:492] T ef990ff6a42c4c72a6e9b39cf26b7e6a P cd94b43daf8042a08b5b91527f42d507: Bootstrap starting.
I20260620 18:42:12.782006 10606 log.cc:824] T ef990ff6a42c4c72a6e9b39cf26b7e6a P 02919670e5434459b173a97c8a0ba38e: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:12.784296 10609 tablet_bootstrap.cc:492] T ef990ff6a42c4c72a6e9b39cf26b7e6a P ca3fc488ac5049adb4badd99378bc814: Bootstrap starting.
I20260620 18:42:12.785341 10607 tablet_bootstrap.cc:654] T ef990ff6a42c4c72a6e9b39cf26b7e6a P cd94b43daf8042a08b5b91527f42d507: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:12.785393 10606 tablet_bootstrap.cc:492] T ef990ff6a42c4c72a6e9b39cf26b7e6a P 02919670e5434459b173a97c8a0ba38e: No bootstrap required, opened a new log
I20260620 18:42:12.785705 10606 ts_tablet_manager.cc:1397] T ef990ff6a42c4c72a6e9b39cf26b7e6a P 02919670e5434459b173a97c8a0ba38e: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 18:42:12.786628 10607 log.cc:824] T ef990ff6a42c4c72a6e9b39cf26b7e6a P cd94b43daf8042a08b5b91527f42d507: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:12.788720 10609 tablet_bootstrap.cc:654] T ef990ff6a42c4c72a6e9b39cf26b7e6a P ca3fc488ac5049adb4badd99378bc814: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:12.789819 10609 log.cc:824] T ef990ff6a42c4c72a6e9b39cf26b7e6a P ca3fc488ac5049adb4badd99378bc814: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:12.792116 10607 tablet_bootstrap.cc:492] T ef990ff6a42c4c72a6e9b39cf26b7e6a P cd94b43daf8042a08b5b91527f42d507: No bootstrap required, opened a new log
I20260620 18:42:12.792445 10607 ts_tablet_manager.cc:1397] T ef990ff6a42c4c72a6e9b39cf26b7e6a P cd94b43daf8042a08b5b91527f42d507: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260620 18:42:12.794626 10606 raft_consensus.cc:348] T ef990ff6a42c4c72a6e9b39cf26b7e6a P 02919670e5434459b173a97c8a0ba38e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02919670e5434459b173a97c8a0ba38e" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 45077 } } peers { permanent_uuid: "cd94b43daf8042a08b5b91527f42d507" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 44991 } } peers { permanent_uuid: "ca3fc488ac5049adb4badd99378bc814" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 45199 } }
I20260620 18:42:12.794879 10609 tablet_bootstrap.cc:492] T ef990ff6a42c4c72a6e9b39cf26b7e6a P ca3fc488ac5049adb4badd99378bc814: No bootstrap required, opened a new log
I20260620 18:42:12.795015 10606 raft_consensus.cc:374] T ef990ff6a42c4c72a6e9b39cf26b7e6a P 02919670e5434459b173a97c8a0ba38e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:12.795130 10609 ts_tablet_manager.cc:1397] T ef990ff6a42c4c72a6e9b39cf26b7e6a P ca3fc488ac5049adb4badd99378bc814: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260620 18:42:12.795146 10606 raft_consensus.cc:729] T ef990ff6a42c4c72a6e9b39cf26b7e6a P 02919670e5434459b173a97c8a0ba38e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02919670e5434459b173a97c8a0ba38e, State: Initialized, Role: FOLLOWER
I20260620 18:42:12.795807 10606 consensus_queue.cc:260] T ef990ff6a42c4c72a6e9b39cf26b7e6a P 02919670e5434459b173a97c8a0ba38e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02919670e5434459b173a97c8a0ba38e" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 45077 } } peers { permanent_uuid: "cd94b43daf8042a08b5b91527f42d507" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 44991 } } peers { permanent_uuid: "ca3fc488ac5049adb4badd99378bc814" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 45199 } }
I20260620 18:42:12.796928 10456 heartbeater.cc:503] Master 127.5.225.190:32883 was elected leader, sending a full tablet report...
I20260620 18:42:12.798995 10606 ts_tablet_manager.cc:1428] T ef990ff6a42c4c72a6e9b39cf26b7e6a P 02919670e5434459b173a97c8a0ba38e: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260620 18:42:12.803108 10607 raft_consensus.cc:348] T ef990ff6a42c4c72a6e9b39cf26b7e6a P cd94b43daf8042a08b5b91527f42d507 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02919670e5434459b173a97c8a0ba38e" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 45077 } } peers { permanent_uuid: "cd94b43daf8042a08b5b91527f42d507" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 44991 } } peers { permanent_uuid: "ca3fc488ac5049adb4badd99378bc814" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 45199 } }
I20260620 18:42:12.803543 10607 raft_consensus.cc:374] T ef990ff6a42c4c72a6e9b39cf26b7e6a P cd94b43daf8042a08b5b91527f42d507 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:12.803637 10607 raft_consensus.cc:729] T ef990ff6a42c4c72a6e9b39cf26b7e6a P cd94b43daf8042a08b5b91527f42d507 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd94b43daf8042a08b5b91527f42d507, State: Initialized, Role: FOLLOWER
I20260620 18:42:12.804265 10607 consensus_queue.cc:260] T ef990ff6a42c4c72a6e9b39cf26b7e6a P cd94b43daf8042a08b5b91527f42d507 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02919670e5434459b173a97c8a0ba38e" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 45077 } } peers { permanent_uuid: "cd94b43daf8042a08b5b91527f42d507" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 44991 } } peers { permanent_uuid: "ca3fc488ac5049adb4badd99378bc814" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 45199 } }
I20260620 18:42:12.805367 10588 heartbeater.cc:503] Master 127.5.225.190:32883 was elected leader, sending a full tablet report...
I20260620 18:42:12.805998 10607 ts_tablet_manager.cc:1428] T ef990ff6a42c4c72a6e9b39cf26b7e6a P cd94b43daf8042a08b5b91527f42d507: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260620 18:42:12.806222 10609 raft_consensus.cc:348] T ef990ff6a42c4c72a6e9b39cf26b7e6a P ca3fc488ac5049adb4badd99378bc814 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02919670e5434459b173a97c8a0ba38e" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 45077 } } peers { permanent_uuid: "cd94b43daf8042a08b5b91527f42d507" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 44991 } } peers { permanent_uuid: "ca3fc488ac5049adb4badd99378bc814" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 45199 } }
I20260620 18:42:12.806612 10609 raft_consensus.cc:374] T ef990ff6a42c4c72a6e9b39cf26b7e6a P ca3fc488ac5049adb4badd99378bc814 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:12.806733 10609 raft_consensus.cc:729] T ef990ff6a42c4c72a6e9b39cf26b7e6a P ca3fc488ac5049adb4badd99378bc814 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca3fc488ac5049adb4badd99378bc814, State: Initialized, Role: FOLLOWER
I20260620 18:42:12.807363 10609 consensus_queue.cc:260] T ef990ff6a42c4c72a6e9b39cf26b7e6a P ca3fc488ac5049adb4badd99378bc814 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02919670e5434459b173a97c8a0ba38e" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 45077 } } peers { permanent_uuid: "cd94b43daf8042a08b5b91527f42d507" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 44991 } } peers { permanent_uuid: "ca3fc488ac5049adb4badd99378bc814" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 45199 } }
I20260620 18:42:12.809070 10609 ts_tablet_manager.cc:1428] T ef990ff6a42c4c72a6e9b39cf26b7e6a P ca3fc488ac5049adb4badd99378bc814: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260620 18:42:12.909228 10324 tablet.cc:2369] T ef990ff6a42c4c72a6e9b39cf26b7e6a P ca3fc488ac5049adb4badd99378bc814: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:42:12.915813 10589 tablet.cc:2369] T ef990ff6a42c4c72a6e9b39cf26b7e6a P cd94b43daf8042a08b5b91527f42d507: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:42:12.920989 10613 raft_consensus.cc:482] T ef990ff6a42c4c72a6e9b39cf26b7e6a P cd94b43daf8042a08b5b91527f42d507 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:42:12.921308 10613 raft_consensus.cc:504] T ef990ff6a42c4c72a6e9b39cf26b7e6a P cd94b43daf8042a08b5b91527f42d507 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02919670e5434459b173a97c8a0ba38e" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 45077 } } peers { permanent_uuid: "cd94b43daf8042a08b5b91527f42d507" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 44991 } } peers { permanent_uuid: "ca3fc488ac5049adb4badd99378bc814" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 45199 } }
I20260620 18:42:12.923054 10613 leader_election.cc:288] T ef990ff6a42c4c72a6e9b39cf26b7e6a P cd94b43daf8042a08b5b91527f42d507 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02919670e5434459b173a97c8a0ba38e (127.5.225.130:45077), ca3fc488ac5049adb4badd99378bc814 (127.5.225.129:45199)
W20260620 18:42:12.925117 10457 tablet.cc:2369] T ef990ff6a42c4c72a6e9b39cf26b7e6a P 02919670e5434459b173a97c8a0ba38e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:42:12.929248 10600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:12.923362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:55605 (local address 127.5.225.130:45077)
0620 18:42:12.923664 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:12.923672 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:12.923695 (+ 23us) server_negotiation.cc:407] Connection header received
0620 18:42:12.923928 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:12.923933 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:12.923993 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:12.924093 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:12.924549 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:12.925627 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:12.927445 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:12.928376 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:12.928546 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:12.928622 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:12.928879 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 18:42:12.928999 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172}
I20260620 18:42:12.929268 10590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:12.923289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.130:45077 (local address 127.5.225.131:55605)
0620 18:42:12.923605 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:12.923626 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:12.923769 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:12.924114 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:12.924122 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:12.924149 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:12.924365 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:12.924376 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:12.925754 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:12.925763 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:12.927268 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:12.927290 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:12.928534 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:12.928539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:12.928723 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:12.928747 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 18:42:12.928852 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 18:42:12.928989 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161}
I20260620 18:42:12.930150 10605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:12.923499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:46561 (local address 127.5.225.129:45199)
0620 18:42:12.923704 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:12.923712 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:12.924370 (+ 658us) server_negotiation.cc:407] Connection header received
0620 18:42:12.924489 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:12.924494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:12.924567 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:12.924633 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:12.925251 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:12.926319 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:12.928341 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:12.929111 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:12.929292 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:12.929361 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:12.929712 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 18:42:12.929896 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:42:12.930284 10411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef990ff6a42c4c72a6e9b39cf26b7e6a" candidate_uuid: "cd94b43daf8042a08b5b91527f42d507" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02919670e5434459b173a97c8a0ba38e" is_pre_election: true
I20260620 18:42:12.930773 10615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:12.923499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:45199 (local address 127.5.225.131:46561)
0620 18:42:12.924259 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:12.924273 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:12.924384 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:12.924675 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:12.924680 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:12.924704 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:12.925049 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:12.925060 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:12.926460 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:12.926472 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:12.928190 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:12.928202 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:12.929313 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:12.929319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:12.929538 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:12.929567 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 18:42:12.930247 (+ 680us) client_negotiation.cc:240] Negotiation successful
0620 18:42:12.930389 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":428,"threads_started":1}
I20260620 18:42:12.930841 10411 raft_consensus.cc:2454] T ef990ff6a42c4c72a6e9b39cf26b7e6a P 02919670e5434459b173a97c8a0ba38e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd94b43daf8042a08b5b91527f42d507 in term 0.
I20260620 18:42:12.931504 10476 leader_election.cc:302] T ef990ff6a42c4c72a6e9b39cf26b7e6a P cd94b43daf8042a08b5b91527f42d507 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02919670e5434459b173a97c8a0ba38e, cd94b43daf8042a08b5b91527f42d507; no voters:
I20260620 18:42:12.931890 10613 raft_consensus.cc:2781] T ef990ff6a42c4c72a6e9b39cf26b7e6a P cd94b43daf8042a08b5b91527f42d507 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:42:12.932061 10613 raft_consensus.cc:482] T ef990ff6a42c4c72a6e9b39cf26b7e6a P cd94b43daf8042a08b5b91527f42d507 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:42:12.931900 10278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef990ff6a42c4c72a6e9b39cf26b7e6a" candidate_uuid: "cd94b43daf8042a08b5b91527f42d507" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca3fc488ac5049adb4badd99378bc814" is_pre_election: true
I20260620 18:42:12.932174 10613 raft_consensus.cc:3037] T ef990ff6a42c4c72a6e9b39cf26b7e6a P cd94b43daf8042a08b5b91527f42d507 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:12.932431 10278 raft_consensus.cc:2454] T ef990ff6a42c4c72a6e9b39cf26b7e6a P ca3fc488ac5049adb4badd99378bc814 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd94b43daf8042a08b5b91527f42d507 in term 0.
I20260620 18:42:12.934692 10613 raft_consensus.cc:504] T ef990ff6a42c4c72a6e9b39cf26b7e6a P cd94b43daf8042a08b5b91527f42d507 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02919670e5434459b173a97c8a0ba38e" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 45077 } } peers { permanent_uuid: "cd94b43daf8042a08b5b91527f42d507" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 44991 } } peers { permanent_uuid: "ca3fc488ac5049adb4badd99378bc814" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 45199 } }
I20260620 18:42:12.935230 10613 leader_election.cc:288] T ef990ff6a42c4c72a6e9b39cf26b7e6a P cd94b43daf8042a08b5b91527f42d507 [CANDIDATE]: Term 1 election: Requested vote from peers 02919670e5434459b173a97c8a0ba38e (127.5.225.130:45077), ca3fc488ac5049adb4badd99378bc814 (127.5.225.129:45199)
I20260620 18:42:12.935585 10411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef990ff6a42c4c72a6e9b39cf26b7e6a" candidate_uuid: "cd94b43daf8042a08b5b91527f42d507" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02919670e5434459b173a97c8a0ba38e"
I20260620 18:42:12.935699 10278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef990ff6a42c4c72a6e9b39cf26b7e6a" candidate_uuid: "cd94b43daf8042a08b5b91527f42d507" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca3fc488ac5049adb4badd99378bc814"
I20260620 18:42:12.935809 10411 raft_consensus.cc:3037] T ef990ff6a42c4c72a6e9b39cf26b7e6a P 02919670e5434459b173a97c8a0ba38e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:12.935922 10278 raft_consensus.cc:3037] T ef990ff6a42c4c72a6e9b39cf26b7e6a P ca3fc488ac5049adb4badd99378bc814 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:12.937990 10411 raft_consensus.cc:2454] T ef990ff6a42c4c72a6e9b39cf26b7e6a P 02919670e5434459b173a97c8a0ba38e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd94b43daf8042a08b5b91527f42d507 in term 1.
I20260620 18:42:12.938000 10278 raft_consensus.cc:2454] T ef990ff6a42c4c72a6e9b39cf26b7e6a P ca3fc488ac5049adb4badd99378bc814 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd94b43daf8042a08b5b91527f42d507 in term 1.
I20260620 18:42:12.938443 10479 leader_election.cc:302] T ef990ff6a42c4c72a6e9b39cf26b7e6a P cd94b43daf8042a08b5b91527f42d507 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca3fc488ac5049adb4badd99378bc814, cd94b43daf8042a08b5b91527f42d507; no voters:
I20260620 18:42:12.938769 10613 raft_consensus.cc:2781] T ef990ff6a42c4c72a6e9b39cf26b7e6a P cd94b43daf8042a08b5b91527f42d507 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:42:12.939209 10613 raft_consensus.cc:686] T ef990ff6a42c4c72a6e9b39cf26b7e6a P cd94b43daf8042a08b5b91527f42d507 [term 1 LEADER]: Becoming Leader. State: Replica: cd94b43daf8042a08b5b91527f42d507, State: Running, Role: LEADER
I20260620 18:42:12.939687 10613 consensus_queue.cc:237] T ef990ff6a42c4c72a6e9b39cf26b7e6a P cd94b43daf8042a08b5b91527f42d507 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02919670e5434459b173a97c8a0ba38e" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 45077 } } peers { permanent_uuid: "cd94b43daf8042a08b5b91527f42d507" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 44991 } } peers { permanent_uuid: "ca3fc488ac5049adb4badd99378bc814" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 45199 } }
I20260620 18:42:12.943953 10134 catalog_manager.cc:5495] T ef990ff6a42c4c72a6e9b39cf26b7e6a P cd94b43daf8042a08b5b91527f42d507 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd94b43daf8042a08b5b91527f42d507 (127.5.225.131). New cstate: current_term: 1 leader_uuid: "cd94b43daf8042a08b5b91527f42d507" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02919670e5434459b173a97c8a0ba38e" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 45077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd94b43daf8042a08b5b91527f42d507" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 44991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca3fc488ac5049adb4badd99378bc814" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 45199 } health_report { overall_health: UNKNOWN } } }
I20260620 18:42:12.985123 10603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:12.980268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37386 (local address 127.5.225.131:44991)
0620 18:42:12.980454 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:12.980459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:12.980607 (+ 148us) server_negotiation.cc:407] Connection header received
0620 18:42:12.980776 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:12.980779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:12.980852 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:12.980971 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:42:12.981289 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:12.981985 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:12.983319 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:12.983706 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:12.983798 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:12.984217 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:42:12.984695 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:42:12.984812 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:12.984877 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 18:42:12.984936 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:42:12.992771 10278 raft_consensus.cc:1261] T ef990ff6a42c4c72a6e9b39cf26b7e6a P ca3fc488ac5049adb4badd99378bc814 [term 1 FOLLOWER]: Refusing update from remote peer cd94b43daf8042a08b5b91527f42d507: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:42:12.992810 10411 raft_consensus.cc:1261] T ef990ff6a42c4c72a6e9b39cf26b7e6a P 02919670e5434459b173a97c8a0ba38e [term 1 FOLLOWER]: Refusing update from remote peer cd94b43daf8042a08b5b91527f42d507: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:42:12.993697 10616 consensus_queue.cc:1035] T ef990ff6a42c4c72a6e9b39cf26b7e6a P cd94b43daf8042a08b5b91527f42d507 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca3fc488ac5049adb4badd99378bc814" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 45199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:42:12.994040 10613 consensus_queue.cc:1035] T ef990ff6a42c4c72a6e9b39cf26b7e6a P cd94b43daf8042a08b5b91527f42d507 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02919670e5434459b173a97c8a0ba38e" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 45077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:42:13.003958 10619 mvcc.cc:205] Tried to move back new op lower bound from 7298993901529571328 to 7298993901324488704. Current Snapshot: MvccSnapshot[applied={T|T < 7298993901529571328}]
I20260620 18:42:13.013652 10620 mvcc.cc:205] Tried to move back new op lower bound from 7298993901529571328 to 7298993901324488704. Current Snapshot: MvccSnapshot[applied={T|T < 7298993901529571328}]
I20260620 18:42:13.018230 10134 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:42:13.018508 10134 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:42:13.027659 10134 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 46465637b04e42999bd9bb839f8d37eb: Sending DeleteTablet for 3 replicas of tablet ef990ff6a42c4c72a6e9b39cf26b7e6a
I20260620 18:42:13.028872 10391 tablet_service.cc:1514] Processing DeleteTablet for tablet ef990ff6a42c4c72a6e9b39cf26b7e6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:42:13 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:55726
I20260620 18:42:13.028874 10523 tablet_service.cc:1514] Processing DeleteTablet for tablet ef990ff6a42c4c72a6e9b39cf26b7e6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:42:13 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:37382
I20260620 18:42:13.030145 10258 tablet_service.cc:1514] Processing DeleteTablet for tablet ef990ff6a42c4c72a6e9b39cf26b7e6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:42:13 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:48936
I20260620 18:42:13.037720 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 10196
I20260620 18:42:13.038532 10631 tablet_replica.cc:330] T ef990ff6a42c4c72a6e9b39cf26b7e6a P ca3fc488ac5049adb4badd99378bc814: stopping tablet replica
I20260620 18:42:13.039266 10631 raft_consensus.cc:2229] T ef990ff6a42c4c72a6e9b39cf26b7e6a P ca3fc488ac5049adb4badd99378bc814 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:42:13.040069 10631 raft_consensus.cc:2258] T ef990ff6a42c4c72a6e9b39cf26b7e6a P ca3fc488ac5049adb4badd99378bc814 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:42:13.041246 10630 tablet_replica.cc:330] T ef990ff6a42c4c72a6e9b39cf26b7e6a P 02919670e5434459b173a97c8a0ba38e: stopping tablet replica
I20260620 18:42:13.041685 10630 raft_consensus.cc:2229] T ef990ff6a42c4c72a6e9b39cf26b7e6a P 02919670e5434459b173a97c8a0ba38e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:42:13.042294 10630 raft_consensus.cc:2258] T ef990ff6a42c4c72a6e9b39cf26b7e6a P 02919670e5434459b173a97c8a0ba38e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:42:13.044246 10630 ts_tablet_manager.cc:1905] T ef990ff6a42c4c72a6e9b39cf26b7e6a P 02919670e5434459b173a97c8a0ba38e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:42:13.044248 10631 ts_tablet_manager.cc:1905] T ef990ff6a42c4c72a6e9b39cf26b7e6a P ca3fc488ac5049adb4badd99378bc814: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:42:13.051913 10630 ts_tablet_manager.cc:1918] T ef990ff6a42c4c72a6e9b39cf26b7e6a P 02919670e5434459b173a97c8a0ba38e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:42:13.052110 10630 log.cc:1196] T ef990ff6a42c4c72a6e9b39cf26b7e6a P 02919670e5434459b173a97c8a0ba38e: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-1/wal/wals/ef990ff6a42c4c72a6e9b39cf26b7e6a
I20260620 18:42:13.052695 10630 ts_tablet_manager.cc:1939] T ef990ff6a42c4c72a6e9b39cf26b7e6a P 02919670e5434459b173a97c8a0ba38e: Deleting consensus metadata
I20260620 18:42:13.054699 10119 catalog_manager.cc:4841] TS 02919670e5434459b173a97c8a0ba38e (127.5.225.130:45077): tablet ef990ff6a42c4c72a6e9b39cf26b7e6a (table test-table [id=15e08cef23864f85933a3bd641c302a8]) successfully deleted
I20260620 18:42:13.058775 10631 ts_tablet_manager.cc:1918] T ef990ff6a42c4c72a6e9b39cf26b7e6a P ca3fc488ac5049adb4badd99378bc814: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:42:13.059103 10631 log.cc:1196] T ef990ff6a42c4c72a6e9b39cf26b7e6a P ca3fc488ac5049adb4badd99378bc814: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-0/wal/wals/ef990ff6a42c4c72a6e9b39cf26b7e6a
Jun 20 18:42:13 dist-test-slave-k6lg krb5kdc[10091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.129@KRBTEST.COM
I20260620 18:42:13.060257 10629 tablet_replica.cc:330] T ef990ff6a42c4c72a6e9b39cf26b7e6a P cd94b43daf8042a08b5b91527f42d507: stopping tablet replica
I20260620 18:42:13.060961 10629 raft_consensus.cc:2229] T ef990ff6a42c4c72a6e9b39cf26b7e6a P cd94b43daf8042a08b5b91527f42d507 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:42:13.061192 10631 ts_tablet_manager.cc:1939] T ef990ff6a42c4c72a6e9b39cf26b7e6a P ca3fc488ac5049adb4badd99378bc814: Deleting consensus metadata
I20260620 18:42:13.061743 10629 raft_consensus.cc:2258] T ef990ff6a42c4c72a6e9b39cf26b7e6a P cd94b43daf8042a08b5b91527f42d507 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:42:13.064850 10629 ts_tablet_manager.cc:1905] T ef990ff6a42c4c72a6e9b39cf26b7e6a P cd94b43daf8042a08b5b91527f42d507: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:42:13.066702 10122 catalog_manager.cc:4841] TS ca3fc488ac5049adb4badd99378bc814 (127.5.225.129:45199): tablet ef990ff6a42c4c72a6e9b39cf26b7e6a (table test-table [id=15e08cef23864f85933a3bd641c302a8]) successfully deleted
I20260620 18:42:13.070134 10605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:13.039276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48942 (local address 127.5.225.129:45199)
0620 18:42:13.039490 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:13.039497 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:13.042395 (+ 2898us) server_negotiation.cc:407] Connection header received
0620 18:42:13.044682 (+ 2287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:13.044689 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:13.044781 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:13.044894 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:13.049063 (+ 4169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:13.050332 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:13.054410 (+ 4078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:13.055171 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:13.055282 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:13.061873 (+ 6591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:13.061889 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:13.061909 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:13.061955 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:13.066276 (+ 4321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:13.067236 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:13.067247 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:13.067251 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:13.067433 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:13.067938 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:13.067944 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:13.067948 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:13.068440 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:13.068755 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:13.069365 (+ 610us) server_negotiation.cc:299] Negotiation successful
0620 18:42:13.069699 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:42:13.071749 10318 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:13.072932 10629 ts_tablet_manager.cc:1918] T ef990ff6a42c4c72a6e9b39cf26b7e6a P cd94b43daf8042a08b5b91527f42d507: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:42:13.073117 10629 log.cc:1196] T ef990ff6a42c4c72a6e9b39cf26b7e6a P cd94b43daf8042a08b5b91527f42d507: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980894474392-6022-0/minicluster-data/ts-2/wal/wals/ef990ff6a42c4c72a6e9b39cf26b7e6a
I20260620 18:42:13.073683 10629 ts_tablet_manager.cc:1939] T ef990ff6a42c4c72a6e9b39cf26b7e6a P cd94b43daf8042a08b5b91527f42d507: Deleting consensus metadata
I20260620 18:42:13.075635 10120 catalog_manager.cc:4841] TS cd94b43daf8042a08b5b91527f42d507 (127.5.225.131:44991): tablet ef990ff6a42c4c72a6e9b39cf26b7e6a (table test-table [id=15e08cef23864f85933a3bd641c302a8]) successfully deleted
I20260620 18:42:13.163959 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 10196
I20260620 18:42:13.185010 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 10329
Jun 20 18:42:13 dist-test-slave-k6lg krb5kdc[10091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.130@KRBTEST.COM
I20260620 18:42:13.204746 10600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:13.186142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55732 (local address 127.5.225.130:45077)
0620 18:42:13.186364 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:13.186370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:13.186390 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:42:13.186452 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:13.186459 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:13.186540 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:13.186652 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:13.188138 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:13.189248 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:13.191239 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:13.191885 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:13.192014 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:13.197030 (+ 5016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:13.197042 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:13.197058 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:13.197095 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:13.200824 (+ 3729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:13.201612 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:13.201621 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:13.201625 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:13.201769 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:13.202202 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:13.202208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:13.202212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:13.202701 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:13.202988 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:13.204097 (+ 1109us) server_negotiation.cc:299] Negotiation successful
0620 18:42:13.204377 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260620 18:42:13.205511 10451 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:13.293958 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 10329
I20260620 18:42:13.316632 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 10461
Jun 20 18:42:13 dist-test-slave-k6lg krb5kdc[10091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.131@KRBTEST.COM
I20260620 18:42:13.332563 10603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:13.317411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37398 (local address 127.5.225.131:44991)
0620 18:42:13.317641 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:13.317648 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:13.317710 (+ 62us) server_negotiation.cc:407] Connection header received
0620 18:42:13.317916 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:13.317921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:13.318002 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:13.318110 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:13.319352 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:13.320155 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:13.321411 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:13.321863 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:13.322006 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:13.326641 (+ 4635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:13.326653 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:13.326668 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:13.326707 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:13.329734 (+ 3027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:13.330472 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:13.330491 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:13.330493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:13.330648 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:13.331034 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:13.331038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:13.331040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:13.331382 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:13.331730 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:13.331991 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 18:42:13.332219 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 18:42:13.333217 10583 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:13.456489 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 10461
I20260620 18:42:13.477186 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 10104
I20260620 18:42:13.478361 10165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:13.585461 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 10104
2026-06-20T18:42:13Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4313 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T18:42:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:42:13Z Disabled control of system clock
I20260620 18:42:13.650774 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:33431
--webserver_interface=127.5.225.190
--webserver_port=0
--builtin_ntp_servers=127.5.225.148:42379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:33431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 18:42:13.982875 10640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:13.983160 10640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:13.983296 10640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:13.990547 10640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:42:13.990651 10640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:13.990729 10640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:13.990849 10640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:42:13.990911 10640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:42:13.999887 10640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:42379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:33431
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:33431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:14.001195 10640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:14.003039 10640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:14.014971 10648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:14.015014 10645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:14.015054 10646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:14.015691 10640 server_base.cc:1034] running on GCE node
I20260620 18:42:14.016578 10640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:14.018132 10640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:14.019459 10640 hybrid_clock.cc:648] HybridClock initialized: now 1781980934019408 us; error 54 us; skew 500 ppm
I20260620 18:42:14.019876 10640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:42:14.022394 10640 webserver.cc:460] Webserver started at http://127.5.225.190:42043/ using document root <none> and password file <none>
I20260620 18:42:14.023109 10640 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:14.023243 10640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:14.023578 10640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:14.026688 10640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "b3e6cfca5c44428ab7c4c336cf065bbc"
format_stamp: "Formatted at 2026-06-20 18:42:14 on dist-test-slave-k6lg"
server_key: "3069ac07617da25e2e6843514ae02360"
server_key_iv: "50e4412473f8b43791e6cf4f357d7b86"
server_key_version: "encryptionkey@0"
I20260620 18:42:14.027518 10640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "b3e6cfca5c44428ab7c4c336cf065bbc"
format_stamp: "Formatted at 2026-06-20 18:42:14 on dist-test-slave-k6lg"
server_key: "3069ac07617da25e2e6843514ae02360"
server_key_iv: "50e4412473f8b43791e6cf4f357d7b86"
server_key_version: "encryptionkey@0"
I20260620 18:42:14.032779 10640 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260620 18:42:14.036379 10654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:14.037935 10640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:42:14.038136 10640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "b3e6cfca5c44428ab7c4c336cf065bbc"
format_stamp: "Formatted at 2026-06-20 18:42:14 on dist-test-slave-k6lg"
server_key: "3069ac07617da25e2e6843514ae02360"
server_key_iv: "50e4412473f8b43791e6cf4f357d7b86"
server_key_version: "encryptionkey@0"
I20260620 18:42:14.038412 10640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:14.044198 10640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:14.044955 10640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:14.045255 10640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:14.063416 10640 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:33431
I20260620 18:42:14.063438 10705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:33431 every 8 connection(s)
I20260620 18:42:14.064790 10640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:42:14.068315 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 10640
I20260620 18:42:14.068514 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:42:14.068792 10706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:14.069235 6022 external_mini_cluster.cc:1428] Setting key 1a43862d4b5788740442697b60ca094a
I20260620 18:42:14.078400 10709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:14.071306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58080 (local address 127.5.225.190:33431)
0620 18:42:14.072382 (+ 1076us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:14.072393 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:14.072435 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:42:14.074736 (+ 2301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:14.074759 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:14.075228 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:14.075641 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:14.076700 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:14.076705 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:14.076723 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:42:14.076737 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:14.076794 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:42:14.076809 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:14.077045 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:14.077190 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 18:42:14.077435 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":388,"threads_started":1}
I20260620 18:42:14.081127 10706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3e6cfca5c44428ab7c4c336cf065bbc: Bootstrap starting.
I20260620 18:42:14.083994 10706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3e6cfca5c44428ab7c4c336cf065bbc: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:14.084985 10706 log.cc:824] T 00000000000000000000000000000000 P b3e6cfca5c44428ab7c4c336cf065bbc: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:14.087798 10706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3e6cfca5c44428ab7c4c336cf065bbc: No bootstrap required, opened a new log
I20260620 18:42:14.095088 10706 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3e6cfca5c44428ab7c4c336cf065bbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3e6cfca5c44428ab7c4c336cf065bbc" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 33431 } }
I20260620 18:42:14.095453 10706 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3e6cfca5c44428ab7c4c336cf065bbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:14.095568 10706 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3e6cfca5c44428ab7c4c336cf065bbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3e6cfca5c44428ab7c4c336cf065bbc, State: Initialized, Role: FOLLOWER
I20260620 18:42:14.096155 10706 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3e6cfca5c44428ab7c4c336cf065bbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3e6cfca5c44428ab7c4c336cf065bbc" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 33431 } }
I20260620 18:42:14.096400 10706 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3e6cfca5c44428ab7c4c336cf065bbc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:42:14.096534 10706 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3e6cfca5c44428ab7c4c336cf065bbc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:42:14.096706 10706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3e6cfca5c44428ab7c4c336cf065bbc [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:14.099294 10706 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3e6cfca5c44428ab7c4c336cf065bbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3e6cfca5c44428ab7c4c336cf065bbc" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 33431 } }
I20260620 18:42:14.099789 10706 leader_election.cc:302] T 00000000000000000000000000000000 P b3e6cfca5c44428ab7c4c336cf065bbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3e6cfca5c44428ab7c4c336cf065bbc; no voters:
I20260620 18:42:14.100382 10706 leader_election.cc:288] T 00000000000000000000000000000000 P b3e6cfca5c44428ab7c4c336cf065bbc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:42:14.100659 10711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3e6cfca5c44428ab7c4c336cf065bbc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:42:14.101385 10711 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3e6cfca5c44428ab7c4c336cf065bbc [term 1 LEADER]: Becoming Leader. State: Replica: b3e6cfca5c44428ab7c4c336cf065bbc, State: Running, Role: LEADER
I20260620 18:42:14.101950 10711 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3e6cfca5c44428ab7c4c336cf065bbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3e6cfca5c44428ab7c4c336cf065bbc" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 33431 } }
I20260620 18:42:14.102741 10706 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3e6cfca5c44428ab7c4c336cf065bbc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:42:14.104933 10712 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3e6cfca5c44428ab7c4c336cf065bbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3e6cfca5c44428ab7c4c336cf065bbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3e6cfca5c44428ab7c4c336cf065bbc" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 33431 } } }
I20260620 18:42:14.104907 10713 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3e6cfca5c44428ab7c4c336cf065bbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3e6cfca5c44428ab7c4c336cf065bbc. Latest consensus state: current_term: 1 leader_uuid: "b3e6cfca5c44428ab7c4c336cf065bbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3e6cfca5c44428ab7c4c336cf065bbc" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 33431 } } }
I20260620 18:42:14.105351 10712 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3e6cfca5c44428ab7c4c336cf065bbc [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:14.105387 10713 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3e6cfca5c44428ab7c4c336cf065bbc [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:14.107678 10718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:42:14.112744 10718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:42:14.120864 10718 catalog_manager.cc:1347] Generated new cluster ID: fd5d529a57f64932a840d5c23b9c4f9c
I20260620 18:42:14.120975 10718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:42:14.141328 10718 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:42:14.141772 10718 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:42:14.149227 10718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3e6cfca5c44428ab7c4c336cf065bbc: Generated new TSK 0
I20260620 18:42:14.150347 10718 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:42:14.167312 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.129:0
--local_ip_for_outbound_sockets=127.5.225.129
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:33431
--builtin_ntp_servers=127.5.225.148:42379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 18:42:14.490674 10730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:14.490950 10730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:14.491102 10730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:14.498298 10730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:14.498389 10730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:14.498608 10730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.129
I20260620 18:42:14.507943 10730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:42379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:33431
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.129
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:14.509222 10730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:14.511004 10730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:14.523748 10735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:14.523799 10738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:14.523722 10736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:14.524832 10730 server_base.cc:1034] running on GCE node
I20260620 18:42:14.525480 10730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:14.526475 10730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:14.527689 10730 hybrid_clock.cc:648] HybridClock initialized: now 1781980934527629 us; error 33 us; skew 500 ppm
I20260620 18:42:14.528004 10730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:42:14.530226 10730 webserver.cc:460] Webserver started at http://127.5.225.129:44495/ using document root <none> and password file <none>
I20260620 18:42:14.530927 10730 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:14.531011 10730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:14.531266 10730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:14.533990 10730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-0/data/instance:
uuid: "8b8d914a55674e79b70135dd51b0b8f6"
format_stamp: "Formatted at 2026-06-20 18:42:14 on dist-test-slave-k6lg"
server_key: "873c7fd218fee51a89a75c394d094490"
server_key_iv: "1bcf566fde0f900aa1b849c36caa9b02"
server_key_version: "encryptionkey@0"
I20260620 18:42:14.534785 10730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance:
uuid: "8b8d914a55674e79b70135dd51b0b8f6"
format_stamp: "Formatted at 2026-06-20 18:42:14 on dist-test-slave-k6lg"
server_key: "873c7fd218fee51a89a75c394d094490"
server_key_iv: "1bcf566fde0f900aa1b849c36caa9b02"
server_key_version: "encryptionkey@0"
I20260620 18:42:14.539731 10730 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:42:14.543021 10744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:14.544298 10730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:42:14.544420 10730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-0/wal
uuid: "8b8d914a55674e79b70135dd51b0b8f6"
format_stamp: "Formatted at 2026-06-20 18:42:14 on dist-test-slave-k6lg"
server_key: "873c7fd218fee51a89a75c394d094490"
server_key_iv: "1bcf566fde0f900aa1b849c36caa9b02"
server_key_version: "encryptionkey@0"
I20260620 18:42:14.544601 10730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:14.550287 10730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:14.550993 10730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:14.551206 10730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:14.552049 10730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:14.553297 10730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:14.553381 10730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:14.553462 10730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:14.553516 10730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:14.586864 10730 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.129:39913
I20260620 18:42:14.586936 10856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.129:39913 every 8 connection(s)
I20260620 18:42:14.588171 10730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
I20260620 18:42:14.596249 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 10730
I20260620 18:42:14.596459 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance
I20260620 18:42:14.596510 10859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:14.590791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:54681 (local address 127.5.225.190:33431)
0620 18:42:14.591248 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:14.591253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:14.592340 (+ 1087us) server_negotiation.cc:407] Connection header received
0620 18:42:14.594237 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:14.594242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:14.594311 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:14.594445 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:14.595768 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:14.595774 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:14.595779 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:42:14.595796 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:14.595844 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:42:14.595856 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:14.595980 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:14.596170 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 18:42:14.596238 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":257,"threads_started":1}
I20260620 18:42:14.597159 6022 external_mini_cluster.cc:1428] Setting key ad1655f832d4cf30a38d761367236eba
I20260620 18:42:14.597488 10860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:14.591284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:33431 (local address 127.5.225.129:54681)
0620 18:42:14.592180 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:14.592216 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:14.593775 (+ 1559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:14.594685 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:14.594694 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:14.594782 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:14.595465 (+ 683us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:42:14.595468 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:14.595505 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:42:14.595592 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:14.595951 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:14.595961 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:14.595968 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 18:42:14.596252 (+ 284us) client_negotiation.cc:240] Negotiation successful
0620 18:42:14.596489 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":273,"threads_started":1}
I20260620 18:42:14.599251 10857 heartbeater.cc:347] Connected to a master server at 127.5.225.190:33431
I20260620 18:42:14.599534 10857 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:14.599926 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.130:0
--local_ip_for_outbound_sockets=127.5.225.130
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:33431
--builtin_ntp_servers=127.5.225.148:42379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 18:42:14.600193 10857 heartbeater.cc:511] Master 127.5.225.190:33431 requested a full tablet report, sending...
I20260620 18:42:14.602120 10671 ts_manager.cc:194] Registered new tserver with Master: 8b8d914a55674e79b70135dd51b0b8f6 (127.5.225.129:39913)
W20260620 18:42:14.919734 10861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:14.919993 10861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:14.920080 10861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:14.927295 10861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:14.927397 10861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:14.927513 10861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.130
I20260620 18:42:14.936537 10861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:42379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:33431
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.130
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:14.937812 10861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:14.939558 10861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:14.952308 10866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:14.952381 10869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:14.952409 10867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:14.953354 10861 server_base.cc:1034] running on GCE node
I20260620 18:42:14.953960 10861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:14.954886 10861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:14.956079 10861 hybrid_clock.cc:648] HybridClock initialized: now 1781980934956042 us; error 29 us; skew 500 ppm
I20260620 18:42:14.956380 10861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:42:14.958595 10861 webserver.cc:460] Webserver started at http://127.5.225.130:44535/ using document root <none> and password file <none>
I20260620 18:42:14.959242 10861 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:14.959348 10861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:14.959669 10861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:14.962431 10861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-1/data/instance:
uuid: "44c5151c71624869b29a7e667f35e6a5"
format_stamp: "Formatted at 2026-06-20 18:42:14 on dist-test-slave-k6lg"
server_key: "12bd5b37e4190d25ad9871023bd6a00f"
server_key_iv: "6896f2aa1937d3dbff285b77066218a5"
server_key_version: "encryptionkey@0"
I20260620 18:42:14.963189 10861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance:
uuid: "44c5151c71624869b29a7e667f35e6a5"
format_stamp: "Formatted at 2026-06-20 18:42:14 on dist-test-slave-k6lg"
server_key: "12bd5b37e4190d25ad9871023bd6a00f"
server_key_iv: "6896f2aa1937d3dbff285b77066218a5"
server_key_version: "encryptionkey@0"
I20260620 18:42:14.967984 10861 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 18:42:14.971683 10875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:14.972913 10861 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 18:42:14.973096 10861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-1/wal
uuid: "44c5151c71624869b29a7e667f35e6a5"
format_stamp: "Formatted at 2026-06-20 18:42:14 on dist-test-slave-k6lg"
server_key: "12bd5b37e4190d25ad9871023bd6a00f"
server_key_iv: "6896f2aa1937d3dbff285b77066218a5"
server_key_version: "encryptionkey@0"
I20260620 18:42:14.973325 10861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:14.978794 10861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:14.979523 10861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:14.979774 10861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:14.980628 10861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:14.981906 10861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:14.982023 10861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:14.982185 10861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:14.982287 10861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 18:42:15.015280 10861 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.130:39969
I20260620 18:42:15.015376 10987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.130:39969 every 8 connection(s)
I20260620 18:42:15.016681 10861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
I20260620 18:42:15.018580 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 10861
I20260620 18:42:15.018755 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance
I20260620 18:42:15.019217 6022 external_mini_cluster.cc:1428] Setting key 3897711dce33270f87b25b2811fc8a25
I20260620 18:42:15.022476 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.131:0
--local_ip_for_outbound_sockets=127.5.225.131
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:33431
--builtin_ntp_servers=127.5.225.148:42379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 18:42:15.027384 10859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:15.019677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:52997 (local address 127.5.225.190:33431)
0620 18:42:15.019896 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:15.019902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:15.021873 (+ 1971us) server_negotiation.cc:407] Connection header received
0620 18:42:15.024610 (+ 2737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:15.024614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:15.024682 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:15.024790 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:15.026408 (+ 1618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:15.026417 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:15.026425 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:42:15.026450 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:15.026491 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:42:15.026508 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:15.026601 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:15.026958 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 18:42:15.027028 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 18:42:15.029008 10990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:15.020350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:33431 (local address 127.5.225.130:52997)
0620 18:42:15.021624 (+ 1274us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:15.021689 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:15.024275 (+ 2586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:15.025056 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:15.025069 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:15.025201 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:15.025997 (+ 796us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:42:15.026003 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:15.026054 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:42:15.026202 (+ 148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:15.026712 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:15.026725 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:15.026735 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 18:42:15.027709 (+ 974us) client_negotiation.cc:240] Negotiation successful
0620 18:42:15.027967 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":367,"threads_started":1}
I20260620 18:42:15.030567 10988 heartbeater.cc:347] Connected to a master server at 127.5.225.190:33431
I20260620 18:42:15.030889 10988 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:15.031586 10988 heartbeater.cc:511] Master 127.5.225.190:33431 requested a full tablet report, sending...
I20260620 18:42:15.033074 10671 ts_manager.cc:194] Registered new tserver with Master: 44c5151c71624869b29a7e667f35e6a5 (127.5.225.130:39969)
W20260620 18:42:15.342890 10991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:15.343184 10991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:15.343338 10991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:15.350606 10991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:15.350732 10991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:15.350908 10991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.131
I20260620 18:42:15.359561 10991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:42379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:33431
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.131
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:15.360807 10991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:15.362571 10991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:15.375362 10997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:15.375254 10999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:15.375325 10996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:15.375998 10991 server_base.cc:1034] running on GCE node
I20260620 18:42:15.376593 10991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:15.377579 10991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:15.378805 10991 hybrid_clock.cc:648] HybridClock initialized: now 1781980935378743 us; error 58 us; skew 500 ppm
I20260620 18:42:15.379117 10991 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:42:15.381264 10991 webserver.cc:460] Webserver started at http://127.5.225.131:46169/ using document root <none> and password file <none>
I20260620 18:42:15.381938 10991 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:15.382059 10991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:15.382418 10991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:15.385133 10991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-2/data/instance:
uuid: "d185cf9d3a184191a37f809f57321237"
format_stamp: "Formatted at 2026-06-20 18:42:15 on dist-test-slave-k6lg"
server_key: "8fe34e6892a4f16c81234277842473af"
server_key_iv: "5bde24266fd2194cd32b5668c669e039"
server_key_version: "encryptionkey@0"
I20260620 18:42:15.385910 10991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance:
uuid: "d185cf9d3a184191a37f809f57321237"
format_stamp: "Formatted at 2026-06-20 18:42:15 on dist-test-slave-k6lg"
server_key: "8fe34e6892a4f16c81234277842473af"
server_key_iv: "5bde24266fd2194cd32b5668c669e039"
server_key_version: "encryptionkey@0"
I20260620 18:42:15.390888 10991 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:42:15.394209 11005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:15.395960 10991 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:42:15.396138 10991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-2/wal
uuid: "d185cf9d3a184191a37f809f57321237"
format_stamp: "Formatted at 2026-06-20 18:42:15 on dist-test-slave-k6lg"
server_key: "8fe34e6892a4f16c81234277842473af"
server_key_iv: "5bde24266fd2194cd32b5668c669e039"
server_key_version: "encryptionkey@0"
I20260620 18:42:15.396413 10991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:15.405303 10991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:15.406035 10991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:15.406334 10991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:15.407167 10991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:15.408509 10991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:15.408594 10991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:15.408679 10991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:15.408735 10991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:15.441062 10991 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.131:45873
I20260620 18:42:15.441148 11117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.131:45873 every 8 connection(s)
I20260620 18:42:15.442508 10991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
I20260620 18:42:15.443189 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 10991
I20260620 18:42:15.443362 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance
I20260620 18:42:15.443967 6022 external_mini_cluster.cc:1428] Setting key a5c96442b88edb46ab09685dae0e5985
I20260620 18:42:15.450944 10859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:15.445366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:59281 (local address 127.5.225.190:33431)
0620 18:42:15.445553 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:15.445559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:15.447245 (+ 1686us) server_negotiation.cc:407] Connection header received
0620 18:42:15.448923 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:15.448927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:15.448977 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:15.449056 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:15.450269 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:15.450273 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:15.450276 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:42:15.450291 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:15.450320 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:42:15.450331 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:15.450430 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:15.450621 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 18:42:15.450685 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:42:15.451805 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:15.445922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:33431 (local address 127.5.225.131:59281)
0620 18:42:15.447075 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:15.447121 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:15.448648 (+ 1527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:15.449256 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:15.449264 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:15.449347 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:15.449960 (+ 613us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:42:15.449962 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:15.449996 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:42:15.450089 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:15.450406 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:15.450412 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:15.450417 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:42:15.450688 (+ 271us) client_negotiation.cc:240] Negotiation successful
0620 18:42:15.450920 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":338,"threads_started":1}
I20260620 18:42:15.453076 11118 heartbeater.cc:347] Connected to a master server at 127.5.225.190:33431
I20260620 18:42:15.453352 11118 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:15.453890 11118 heartbeater.cc:511] Master 127.5.225.190:33431 requested a full tablet report, sending...
I20260620 18:42:15.455196 10671 ts_manager.cc:194] Registered new tserver with Master: d185cf9d3a184191a37f809f57321237 (127.5.225.131:45873)
I20260620 18:42:15.459439 6022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:42:15.466151 10859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:15.463826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58084 (local address 127.5.225.190:33431)
0620 18:42:15.464003 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:15.464012 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:15.464460 (+ 448us) server_negotiation.cc:407] Connection header received
0620 18:42:15.464579 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:15.464583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:15.464631 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:15.464712 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:15.465574 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:15.465578 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:15.465581 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:42:15.465595 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:15.465620 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:42:15.465631 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:15.465725 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:15.465793 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 18:42:15.465834 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 18:42:15.469906 10671 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:42:15.476514 10671 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:42:15.508139 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:15.488195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.130:39969 (local address 127.0.0.1:50940)
0620 18:42:15.500545 (+ 12350us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:15.500585 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:15.500721 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:15.505531 (+ 4810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:15.505540 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:15.505663 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:15.506737 (+ 1074us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:42:15.506743 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:15.506781 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:42:15.506881 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:15.507417 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:15.507430 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:15.507437 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 18:42:15.507658 (+ 221us) client_negotiation.cc:240] Negotiation successful
0620 18:42:15.507740 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12097,"thread_start_us":18248,"threads_started":1}
I20260620 18:42:15.508109 11132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:15.488559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50940 (local address 127.5.225.130:39969)
0620 18:42:15.502795 (+ 14236us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:15.502809 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:15.502847 (+ 38us) server_negotiation.cc:407] Connection header received
0620 18:42:15.502935 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:15.502946 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:15.504164 (+ 1218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:15.504347 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:15.507073 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:15.507082 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:15.507099 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:42:15.507120 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:15.507178 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:42:15.507202 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:15.507367 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:15.507622 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 18:42:15.507729 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14014,"thread_start_us":13876,"threads_started":1}
I20260620 18:42:15.510282 11135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:15.489970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45962 (local address 127.5.225.131:45873)
0620 18:42:15.507824 (+ 17854us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:15.507832 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:15.507866 (+ 34us) server_negotiation.cc:407] Connection header received
0620 18:42:15.507939 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:15.507948 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:15.508119 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:15.508304 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:15.509443 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:15.509449 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:15.509457 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:42:15.509481 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:15.509519 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:42:15.509532 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:15.509732 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:15.509796 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 18:42:15.509887 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17084,"thread_start_us":16853,"threads_started":1}
I20260620 18:42:15.511052 11133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:15.488965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51320 (local address 127.5.225.129:39913)
0620 18:42:15.507536 (+ 18571us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:15.507543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:15.507566 (+ 23us) server_negotiation.cc:407] Connection header received
0620 18:42:15.507620 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:15.507625 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:15.507854 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:15.508006 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:15.509732 (+ 1726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:15.509739 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:15.509751 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:42:15.509776 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:15.509831 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:42:15.509848 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:15.509968 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:15.510623 (+ 655us) server_negotiation.cc:299] Negotiation successful
0620 18:42:15.510729 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18418,"thread_start_us":18305,"threads_started":1}
I20260620 18:42:15.511603 11130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:15.489695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.131:45873 (local address 127.0.0.1:45962)
0620 18:42:15.507239 (+ 17544us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:15.507257 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:15.507371 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:15.508353 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:15.508360 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:15.508387 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:15.509235 (+ 848us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:42:15.509240 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:15.509259 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:42:15.509306 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:15.509647 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:15.509652 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:15.509657 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:42:15.509780 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 18:42:15.509830 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17436,"thread_start_us":16487,"threads_started":1}
I20260620 18:42:15.511787 11134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:15.488263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:39913 (local address 127.0.0.1:51320)
0620 18:42:15.504823 (+ 16560us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:15.504842 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:15.504943 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:15.508807 (+ 3864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:15.508813 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:15.508835 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:15.509517 (+ 682us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:42:15.509520 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:15.509533 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:42:15.509571 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:15.510022 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:15.510055 (+ 33us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:15.510061 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 18:42:15.510247 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 18:42:15.510286 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16418,"thread_start_us":11865,"threads_started":1}
I20260620 18:42:15.513278 10923 tablet_service.cc:1467] Processing CreateTablet for tablet 2511ba62d3d841e18148f6afaacf922e (DEFAULT_TABLE table=test-table [id=0f39bc4d2e5e46f9bbebe6436b56c68f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:42:15.514822 10923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2511ba62d3d841e18148f6afaacf922e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:15.514649 11053 tablet_service.cc:1467] Processing CreateTablet for tablet 2511ba62d3d841e18148f6afaacf922e (DEFAULT_TABLE table=test-table [id=0f39bc4d2e5e46f9bbebe6436b56c68f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:42:15.515213 10792 tablet_service.cc:1467] Processing CreateTablet for tablet 2511ba62d3d841e18148f6afaacf922e (DEFAULT_TABLE table=test-table [id=0f39bc4d2e5e46f9bbebe6436b56c68f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:42:15.515821 11053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2511ba62d3d841e18148f6afaacf922e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:15.516455 10792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2511ba62d3d841e18148f6afaacf922e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:15.527691 11136 tablet_bootstrap.cc:492] T 2511ba62d3d841e18148f6afaacf922e P d185cf9d3a184191a37f809f57321237: Bootstrap starting.
I20260620 18:42:15.529599 11137 tablet_bootstrap.cc:492] T 2511ba62d3d841e18148f6afaacf922e P 44c5151c71624869b29a7e667f35e6a5: Bootstrap starting.
I20260620 18:42:15.531184 11138 tablet_bootstrap.cc:492] T 2511ba62d3d841e18148f6afaacf922e P 8b8d914a55674e79b70135dd51b0b8f6: Bootstrap starting.
I20260620 18:42:15.532757 11137 tablet_bootstrap.cc:654] T 2511ba62d3d841e18148f6afaacf922e P 44c5151c71624869b29a7e667f35e6a5: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:15.532768 11136 tablet_bootstrap.cc:654] T 2511ba62d3d841e18148f6afaacf922e P d185cf9d3a184191a37f809f57321237: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:15.533922 11138 tablet_bootstrap.cc:654] T 2511ba62d3d841e18148f6afaacf922e P 8b8d914a55674e79b70135dd51b0b8f6: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:15.534008 11136 log.cc:824] T 2511ba62d3d841e18148f6afaacf922e P d185cf9d3a184191a37f809f57321237: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:15.534008 11137 log.cc:824] T 2511ba62d3d841e18148f6afaacf922e P 44c5151c71624869b29a7e667f35e6a5: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:15.535152 11138 log.cc:824] T 2511ba62d3d841e18148f6afaacf922e P 8b8d914a55674e79b70135dd51b0b8f6: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:15.536948 11137 tablet_bootstrap.cc:492] T 2511ba62d3d841e18148f6afaacf922e P 44c5151c71624869b29a7e667f35e6a5: No bootstrap required, opened a new log
I20260620 18:42:15.537114 11136 tablet_bootstrap.cc:492] T 2511ba62d3d841e18148f6afaacf922e P d185cf9d3a184191a37f809f57321237: No bootstrap required, opened a new log
I20260620 18:42:15.537242 11137 ts_tablet_manager.cc:1397] T 2511ba62d3d841e18148f6afaacf922e P 44c5151c71624869b29a7e667f35e6a5: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260620 18:42:15.537386 11136 ts_tablet_manager.cc:1397] T 2511ba62d3d841e18148f6afaacf922e P d185cf9d3a184191a37f809f57321237: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260620 18:42:15.537918 11138 tablet_bootstrap.cc:492] T 2511ba62d3d841e18148f6afaacf922e P 8b8d914a55674e79b70135dd51b0b8f6: No bootstrap required, opened a new log
I20260620 18:42:15.538192 11138 ts_tablet_manager.cc:1397] T 2511ba62d3d841e18148f6afaacf922e P 8b8d914a55674e79b70135dd51b0b8f6: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260620 18:42:15.544946 11137 raft_consensus.cc:348] T 2511ba62d3d841e18148f6afaacf922e P 44c5151c71624869b29a7e667f35e6a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c5151c71624869b29a7e667f35e6a5" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39969 } } peers { permanent_uuid: "8b8d914a55674e79b70135dd51b0b8f6" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 39913 } } peers { permanent_uuid: "d185cf9d3a184191a37f809f57321237" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 45873 } }
I20260620 18:42:15.545305 11137 raft_consensus.cc:374] T 2511ba62d3d841e18148f6afaacf922e P 44c5151c71624869b29a7e667f35e6a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:15.545395 11137 raft_consensus.cc:729] T 2511ba62d3d841e18148f6afaacf922e P 44c5151c71624869b29a7e667f35e6a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44c5151c71624869b29a7e667f35e6a5, State: Initialized, Role: FOLLOWER
I20260620 18:42:15.546001 11137 consensus_queue.cc:260] T 2511ba62d3d841e18148f6afaacf922e P 44c5151c71624869b29a7e667f35e6a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c5151c71624869b29a7e667f35e6a5" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39969 } } peers { permanent_uuid: "8b8d914a55674e79b70135dd51b0b8f6" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 39913 } } peers { permanent_uuid: "d185cf9d3a184191a37f809f57321237" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 45873 } }
I20260620 18:42:15.547045 10988 heartbeater.cc:503] Master 127.5.225.190:33431 was elected leader, sending a full tablet report...
I20260620 18:42:15.547712 11137 ts_tablet_manager.cc:1428] T 2511ba62d3d841e18148f6afaacf922e P 44c5151c71624869b29a7e667f35e6a5: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260620 18:42:15.547739 11136 raft_consensus.cc:348] T 2511ba62d3d841e18148f6afaacf922e P d185cf9d3a184191a37f809f57321237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c5151c71624869b29a7e667f35e6a5" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39969 } } peers { permanent_uuid: "8b8d914a55674e79b70135dd51b0b8f6" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 39913 } } peers { permanent_uuid: "d185cf9d3a184191a37f809f57321237" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 45873 } }
I20260620 18:42:15.548138 11136 raft_consensus.cc:374] T 2511ba62d3d841e18148f6afaacf922e P d185cf9d3a184191a37f809f57321237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:15.548243 11136 raft_consensus.cc:729] T 2511ba62d3d841e18148f6afaacf922e P d185cf9d3a184191a37f809f57321237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d185cf9d3a184191a37f809f57321237, State: Initialized, Role: FOLLOWER
I20260620 18:42:15.548831 11136 consensus_queue.cc:260] T 2511ba62d3d841e18148f6afaacf922e P d185cf9d3a184191a37f809f57321237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c5151c71624869b29a7e667f35e6a5" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39969 } } peers { permanent_uuid: "8b8d914a55674e79b70135dd51b0b8f6" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 39913 } } peers { permanent_uuid: "d185cf9d3a184191a37f809f57321237" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 45873 } }
I20260620 18:42:15.549664 11138 raft_consensus.cc:348] T 2511ba62d3d841e18148f6afaacf922e P 8b8d914a55674e79b70135dd51b0b8f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c5151c71624869b29a7e667f35e6a5" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39969 } } peers { permanent_uuid: "8b8d914a55674e79b70135dd51b0b8f6" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 39913 } } peers { permanent_uuid: "d185cf9d3a184191a37f809f57321237" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 45873 } }
I20260620 18:42:15.550280 11138 raft_consensus.cc:374] T 2511ba62d3d841e18148f6afaacf922e P 8b8d914a55674e79b70135dd51b0b8f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:15.550395 11138 raft_consensus.cc:729] T 2511ba62d3d841e18148f6afaacf922e P 8b8d914a55674e79b70135dd51b0b8f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8d914a55674e79b70135dd51b0b8f6, State: Initialized, Role: FOLLOWER
I20260620 18:42:15.550534 11136 ts_tablet_manager.cc:1428] T 2511ba62d3d841e18148f6afaacf922e P d185cf9d3a184191a37f809f57321237: Time spent starting tablet: real 0.013s user 0.011s sys 0.004s
I20260620 18:42:15.550904 11118 heartbeater.cc:503] Master 127.5.225.190:33431 was elected leader, sending a full tablet report...
I20260620 18:42:15.551071 11138 consensus_queue.cc:260] T 2511ba62d3d841e18148f6afaacf922e P 8b8d914a55674e79b70135dd51b0b8f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c5151c71624869b29a7e667f35e6a5" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39969 } } peers { permanent_uuid: "8b8d914a55674e79b70135dd51b0b8f6" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 39913 } } peers { permanent_uuid: "d185cf9d3a184191a37f809f57321237" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 45873 } }
I20260620 18:42:15.552057 10857 heartbeater.cc:503] Master 127.5.225.190:33431 was elected leader, sending a full tablet report...
I20260620 18:42:15.553614 11138 ts_tablet_manager.cc:1428] T 2511ba62d3d841e18148f6afaacf922e P 8b8d914a55674e79b70135dd51b0b8f6: Time spent starting tablet: real 0.015s user 0.009s sys 0.003s
W20260620 18:42:15.590436 10858 tablet.cc:2369] T 2511ba62d3d841e18148f6afaacf922e P 8b8d914a55674e79b70135dd51b0b8f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:42:15.659595 11144 raft_consensus.cc:482] T 2511ba62d3d841e18148f6afaacf922e P 8b8d914a55674e79b70135dd51b0b8f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:42:15.659963 11144 raft_consensus.cc:504] T 2511ba62d3d841e18148f6afaacf922e P 8b8d914a55674e79b70135dd51b0b8f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c5151c71624869b29a7e667f35e6a5" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39969 } } peers { permanent_uuid: "8b8d914a55674e79b70135dd51b0b8f6" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 39913 } } peers { permanent_uuid: "d185cf9d3a184191a37f809f57321237" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 45873 } }
I20260620 18:42:15.661710 11144 leader_election.cc:288] T 2511ba62d3d841e18148f6afaacf922e P 8b8d914a55674e79b70135dd51b0b8f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44c5151c71624869b29a7e667f35e6a5 (127.5.225.130:39969), d185cf9d3a184191a37f809f57321237 (127.5.225.131:45873)
I20260620 18:42:15.664934 11135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:15.662152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:40847 (local address 127.5.225.131:45873)
0620 18:42:15.662363 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:15.662370 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:15.662928 (+ 558us) server_negotiation.cc:407] Connection header received
0620 18:42:15.663123 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:15.663129 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:15.663188 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:15.663294 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:15.664332 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:15.664340 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:15.664349 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:42:15.664364 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:15.664394 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:42:15.664406 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:15.664532 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:15.664605 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 18:42:15.664668 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 18:42:15.665150 11145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:15.662078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.131:45873 (local address 127.5.225.129:40847)
0620 18:42:15.662803 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:15.662818 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:15.662975 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:15.663320 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:15.663324 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:15.663348 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:15.664100 (+ 752us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:42:15.664109 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:15.664127 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:42:15.664177 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:15.664499 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:15.664503 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:15.664508 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:42:15.664663 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 18:42:15.664711 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":490,"threads_started":1}
I20260620 18:42:15.665330 11132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:15.662336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:37963 (local address 127.5.225.130:39969)
0620 18:42:15.662551 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:15.662560 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:15.663424 (+ 864us) server_negotiation.cc:407] Connection header received
0620 18:42:15.663609 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:15.663615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:15.663682 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:15.663784 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:15.664741 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:15.664746 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:15.664752 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:42:15.664771 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:15.664798 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:42:15.664809 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:15.664932 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:15.665012 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 18:42:15.665085 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 18:42:15.665745 11146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:15.662140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.130:39969 (local address 127.5.225.129:37963)
0620 18:42:15.663318 (+ 1178us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:15.663333 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:15.663468 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:15.663824 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:15.663827 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:15.663843 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:15.664506 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:42:15.664509 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:15.664521 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:42:15.664590 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:15.664900 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:15.664904 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:15.664910 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 18:42:15.665063 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 18:42:15.665112 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1054,"thread_start_us":779,"threads_started":1}
I20260620 18:42:15.666321 11073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2511ba62d3d841e18148f6afaacf922e" candidate_uuid: "8b8d914a55674e79b70135dd51b0b8f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d185cf9d3a184191a37f809f57321237" is_pre_election: true
I20260620 18:42:15.666821 11073 raft_consensus.cc:2454] T 2511ba62d3d841e18148f6afaacf922e P d185cf9d3a184191a37f809f57321237 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b8d914a55674e79b70135dd51b0b8f6 in term 0.
I20260620 18:42:15.666775 10943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2511ba62d3d841e18148f6afaacf922e" candidate_uuid: "8b8d914a55674e79b70135dd51b0b8f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44c5151c71624869b29a7e667f35e6a5" is_pre_election: true
I20260620 18:42:15.667193 10943 raft_consensus.cc:2454] T 2511ba62d3d841e18148f6afaacf922e P 44c5151c71624869b29a7e667f35e6a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b8d914a55674e79b70135dd51b0b8f6 in term 0.
I20260620 18:42:15.667470 10745 leader_election.cc:302] T 2511ba62d3d841e18148f6afaacf922e P 8b8d914a55674e79b70135dd51b0b8f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b8d914a55674e79b70135dd51b0b8f6, d185cf9d3a184191a37f809f57321237; no voters:
I20260620 18:42:15.667906 11144 raft_consensus.cc:2781] T 2511ba62d3d841e18148f6afaacf922e P 8b8d914a55674e79b70135dd51b0b8f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:42:15.668046 11144 raft_consensus.cc:482] T 2511ba62d3d841e18148f6afaacf922e P 8b8d914a55674e79b70135dd51b0b8f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:42:15.668156 11144 raft_consensus.cc:3037] T 2511ba62d3d841e18148f6afaacf922e P 8b8d914a55674e79b70135dd51b0b8f6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:15.670274 11144 raft_consensus.cc:504] T 2511ba62d3d841e18148f6afaacf922e P 8b8d914a55674e79b70135dd51b0b8f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c5151c71624869b29a7e667f35e6a5" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39969 } } peers { permanent_uuid: "8b8d914a55674e79b70135dd51b0b8f6" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 39913 } } peers { permanent_uuid: "d185cf9d3a184191a37f809f57321237" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 45873 } }
I20260620 18:42:15.670820 11144 leader_election.cc:288] T 2511ba62d3d841e18148f6afaacf922e P 8b8d914a55674e79b70135dd51b0b8f6 [CANDIDATE]: Term 1 election: Requested vote from peers 44c5151c71624869b29a7e667f35e6a5 (127.5.225.130:39969), d185cf9d3a184191a37f809f57321237 (127.5.225.131:45873)
I20260620 18:42:15.671229 10943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2511ba62d3d841e18148f6afaacf922e" candidate_uuid: "8b8d914a55674e79b70135dd51b0b8f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44c5151c71624869b29a7e667f35e6a5"
I20260620 18:42:15.671273 11073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2511ba62d3d841e18148f6afaacf922e" candidate_uuid: "8b8d914a55674e79b70135dd51b0b8f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d185cf9d3a184191a37f809f57321237"
I20260620 18:42:15.671434 10943 raft_consensus.cc:3037] T 2511ba62d3d841e18148f6afaacf922e P 44c5151c71624869b29a7e667f35e6a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:15.671442 11073 raft_consensus.cc:3037] T 2511ba62d3d841e18148f6afaacf922e P d185cf9d3a184191a37f809f57321237 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:15.674647 10943 raft_consensus.cc:2454] T 2511ba62d3d841e18148f6afaacf922e P 44c5151c71624869b29a7e667f35e6a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b8d914a55674e79b70135dd51b0b8f6 in term 1.
I20260620 18:42:15.674638 11073 raft_consensus.cc:2454] T 2511ba62d3d841e18148f6afaacf922e P d185cf9d3a184191a37f809f57321237 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b8d914a55674e79b70135dd51b0b8f6 in term 1.
I20260620 18:42:15.675024 10745 leader_election.cc:302] T 2511ba62d3d841e18148f6afaacf922e P 8b8d914a55674e79b70135dd51b0b8f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b8d914a55674e79b70135dd51b0b8f6, d185cf9d3a184191a37f809f57321237; no voters:
I20260620 18:42:15.675280 11144 raft_consensus.cc:2781] T 2511ba62d3d841e18148f6afaacf922e P 8b8d914a55674e79b70135dd51b0b8f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:42:15.675688 11144 raft_consensus.cc:686] T 2511ba62d3d841e18148f6afaacf922e P 8b8d914a55674e79b70135dd51b0b8f6 [term 1 LEADER]: Becoming Leader. State: Replica: 8b8d914a55674e79b70135dd51b0b8f6, State: Running, Role: LEADER
I20260620 18:42:15.676242 11144 consensus_queue.cc:237] T 2511ba62d3d841e18148f6afaacf922e P 8b8d914a55674e79b70135dd51b0b8f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c5151c71624869b29a7e667f35e6a5" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39969 } } peers { permanent_uuid: "8b8d914a55674e79b70135dd51b0b8f6" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 39913 } } peers { permanent_uuid: "d185cf9d3a184191a37f809f57321237" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 45873 } }
I20260620 18:42:15.680657 10670 catalog_manager.cc:5495] T 2511ba62d3d841e18148f6afaacf922e P 8b8d914a55674e79b70135dd51b0b8f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b8d914a55674e79b70135dd51b0b8f6 (127.5.225.129). New cstate: current_term: 1 leader_uuid: "8b8d914a55674e79b70135dd51b0b8f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c5151c71624869b29a7e667f35e6a5" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b8d914a55674e79b70135dd51b0b8f6" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 39913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d185cf9d3a184191a37f809f57321237" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 45873 } health_report { overall_health: UNKNOWN } } }
W20260620 18:42:15.694607 11119 tablet.cc:2369] T 2511ba62d3d841e18148f6afaacf922e P d185cf9d3a184191a37f809f57321237: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:42:15.736724 11133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:15.734658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51330 (local address 127.5.225.129:39913)
0620 18:42:15.734852 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:15.734858 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:15.734879 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:42:15.734987 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:15.734993 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:15.735082 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:15.735206 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:15.736072 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:15.736078 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:15.736083 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:42:15.736101 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:15.736147 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:42:15.736159 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:15.736279 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:15.736393 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 18:42:15.736453 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 18:42:15.744963 11073 raft_consensus.cc:1261] T 2511ba62d3d841e18148f6afaacf922e P d185cf9d3a184191a37f809f57321237 [term 1 FOLLOWER]: Refusing update from remote peer 8b8d914a55674e79b70135dd51b0b8f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:42:15.745077 10943 raft_consensus.cc:1261] T 2511ba62d3d841e18148f6afaacf922e P 44c5151c71624869b29a7e667f35e6a5 [term 1 FOLLOWER]: Refusing update from remote peer 8b8d914a55674e79b70135dd51b0b8f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:42:15.745987 11144 consensus_queue.cc:1035] T 2511ba62d3d841e18148f6afaacf922e P 8b8d914a55674e79b70135dd51b0b8f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d185cf9d3a184191a37f809f57321237" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 45873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:42:15.746330 11147 consensus_queue.cc:1035] T 2511ba62d3d841e18148f6afaacf922e P 8b8d914a55674e79b70135dd51b0b8f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44c5151c71624869b29a7e667f35e6a5" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:42:15.771796 10670 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:42:15.772086 10670 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:42:15.775605 10670 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b3e6cfca5c44428ab7c4c336cf065bbc: Sending DeleteTablet for 3 replicas of tablet 2511ba62d3d841e18148f6afaacf922e
I20260620 18:42:15.776659 10923 tablet_service.cc:1514] Processing DeleteTablet for tablet 2511ba62d3d841e18148f6afaacf922e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:42:15 UTC) from {username='slave'} at 127.0.0.1:50940
I20260620 18:42:15.776782 11053 tablet_service.cc:1514] Processing DeleteTablet for tablet 2511ba62d3d841e18148f6afaacf922e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:42:15 UTC) from {username='slave'} at 127.0.0.1:45962
I20260620 18:42:15.777801 10792 tablet_service.cc:1514] Processing DeleteTablet for tablet 2511ba62d3d841e18148f6afaacf922e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:42:15 UTC) from {username='slave'} at 127.0.0.1:51320
I20260620 18:42:15.778293 11159 tablet_replica.cc:330] T 2511ba62d3d841e18148f6afaacf922e P 44c5151c71624869b29a7e667f35e6a5: stopping tablet replica
I20260620 18:42:15.778666 11159 raft_consensus.cc:2229] T 2511ba62d3d841e18148f6afaacf922e P 44c5151c71624869b29a7e667f35e6a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:42:15.778724 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 10730
I20260620 18:42:15.779124 11159 raft_consensus.cc:2258] T 2511ba62d3d841e18148f6afaacf922e P 44c5151c71624869b29a7e667f35e6a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:42:15.779784 11161 tablet_replica.cc:330] T 2511ba62d3d841e18148f6afaacf922e P 8b8d914a55674e79b70135dd51b0b8f6: stopping tablet replica
I20260620 18:42:15.780139 11161 raft_consensus.cc:2229] T 2511ba62d3d841e18148f6afaacf922e P 8b8d914a55674e79b70135dd51b0b8f6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:42:15.780727 11161 raft_consensus.cc:2258] T 2511ba62d3d841e18148f6afaacf922e P 8b8d914a55674e79b70135dd51b0b8f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:42:15.781214 11159 ts_tablet_manager.cc:1905] T 2511ba62d3d841e18148f6afaacf922e P 44c5151c71624869b29a7e667f35e6a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:42:15.790463 11159 ts_tablet_manager.cc:1918] T 2511ba62d3d841e18148f6afaacf922e P 44c5151c71624869b29a7e667f35e6a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:42:15.790699 11159 log.cc:1196] T 2511ba62d3d841e18148f6afaacf922e P 44c5151c71624869b29a7e667f35e6a5: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-1/wal/wals/2511ba62d3d841e18148f6afaacf922e
I20260620 18:42:15.791306 11159 ts_tablet_manager.cc:1939] T 2511ba62d3d841e18148f6afaacf922e P 44c5151c71624869b29a7e667f35e6a5: Deleting consensus metadata
I20260620 18:42:15.792152 11160 tablet_replica.cc:330] T 2511ba62d3d841e18148f6afaacf922e P d185cf9d3a184191a37f809f57321237: stopping tablet replica
I20260620 18:42:15.792270 11133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:15.781284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51346 (local address 127.5.225.129:39913)
0620 18:42:15.781442 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:15.781447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:15.789868 (+ 8421us) server_negotiation.cc:407] Connection header received
0620 18:42:15.790077 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:15.790083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:15.790149 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:15.790283 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:15.791463 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:15.791469 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:15.791475 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:42:15.791515 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:15.791553 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:42:15.791569 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:15.791664 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:15.791936 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 18:42:15.791995 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:42:15.792553 11160 raft_consensus.cc:2229] T 2511ba62d3d841e18148f6afaacf922e P d185cf9d3a184191a37f809f57321237 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:42:15.792999 11160 raft_consensus.cc:2258] T 2511ba62d3d841e18148f6afaacf922e P d185cf9d3a184191a37f809f57321237 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:42:15.793182 10656 catalog_manager.cc:4841] TS 44c5151c71624869b29a7e667f35e6a5 (127.5.225.130:39969): tablet 2511ba62d3d841e18148f6afaacf922e (table test-table [id=0f39bc4d2e5e46f9bbebe6436b56c68f]) successfully deleted
I20260620 18:42:15.793452 10852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:15.795212 11160 ts_tablet_manager.cc:1905] T 2511ba62d3d841e18148f6afaacf922e P d185cf9d3a184191a37f809f57321237: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:42:15.801206 11160 ts_tablet_manager.cc:1918] T 2511ba62d3d841e18148f6afaacf922e P d185cf9d3a184191a37f809f57321237: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:42:15.801357 11160 log.cc:1196] T 2511ba62d3d841e18148f6afaacf922e P d185cf9d3a184191a37f809f57321237: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980894474392-6022-0/minicluster-data/ts-2/wal/wals/2511ba62d3d841e18148f6afaacf922e
I20260620 18:42:15.801889 11160 ts_tablet_manager.cc:1939] T 2511ba62d3d841e18148f6afaacf922e P d185cf9d3a184191a37f809f57321237: Deleting consensus metadata
I20260620 18:42:15.803464 10655 catalog_manager.cc:4841] TS d185cf9d3a184191a37f809f57321237 (127.5.225.131:45873): tablet 2511ba62d3d841e18148f6afaacf922e (table test-table [id=0f39bc4d2e5e46f9bbebe6436b56c68f]) successfully deleted
I20260620 18:42:15.888621 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 10730
I20260620 18:42:15.906376 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 10861
W20260620 18:42:15.906276 10657 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.225.129:39913 (error 108)
I20260620 18:42:15.908224 11134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:15.907764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:39913 (local address 127.0.0.1:51358)
0620 18:42:15.907917 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:15.908043 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.129:39913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 18:42:15.908437 10657 catalog_manager.cc:4584] TS 8b8d914a55674e79b70135dd51b0b8f6 (127.5.225.129:39913): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2511ba62d3d841e18148f6afaacf922e: Network error: Client connection negotiation failed: client connection to 127.5.225.129:39913: connect: Connection refused (error 111)
I20260620 18:42:15.909430 11132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:15.907136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50946 (local address 127.5.225.130:39969)
0620 18:42:15.907315 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:15.907321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:15.907411 (+ 90us) server_negotiation.cc:407] Connection header received
0620 18:42:15.907648 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:15.907653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:15.907764 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:15.907876 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:15.908828 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:15.908832 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:15.908835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:42:15.908849 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:15.908880 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:42:15.908892 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:15.908992 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:15.909096 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 18:42:15.909137 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:42:15.910383 10983 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:15.962361 11134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:15.961786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:39913 (local address 127.0.0.1:51370)
0620 18:42:15.962017 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:15.962139 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.129:39913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 18:42:15.963414 11134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:15.963033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:39913 (local address 127.0.0.1:51372)
0620 18:42:15.963188 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:15.963271 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.129:39913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 18:42:15.997879 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 10861
I20260620 18:42:16.014683 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 10991
I20260620 18:42:16.017333 11135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:16.015496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45976 (local address 127.5.225.131:45873)
0620 18:42:16.015667 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:16.015672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:16.015690 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:42:16.015827 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:16.015830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:16.015907 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:16.015997 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:16.016818 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:16.016823 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:16.016826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:42:16.016841 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:16.016872 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:42:16.016884 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:16.016989 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:16.017046 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 18:42:16.017091 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 18:42:16.017949 11113 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:16.023746 11134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:16.023348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:39913 (local address 127.0.0.1:51388)
0620 18:42:16.023530 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:16.023611 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.129:39913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 18:42:16.024789 11134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:16.024478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:39913 (local address 127.0.0.1:51396)
0620 18:42:16.024602 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:16.024676 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.129:39913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 18:42:16.105365 11134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:16.104885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:39913 (local address 127.0.0.1:51412)
0620 18:42:16.105093 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:16.105205 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.129:39913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 18:42:16.106549 11134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:16.106057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:39913 (local address 127.0.0.1:51416)
0620 18:42:16.106271 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:16.106385 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.129:39913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 18:42:16.108228 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 10991
I20260620 18:42:16.124162 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 10640
I20260620 18:42:16.125407 10701 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:16.235702 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 10640
2026-06-20T18:42:16Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2649 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 18:42:16.276603 6022 mini_oidc.cc:152] Starting JWKS server
I20260620 18:42:16.279223 6022 webserver.cc:460] Webserver started at https://127.0.0.1:44607/ using document root <none> and password file <none>
I20260620 18:42:16.279495 6022 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 18:42:16.280092 6022 webserver.cc:460] Webserver started at http://127.0.0.1:38923/ using document root <none> and password file <none>
2026-06-20T18:42:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:42:16Z Disabled control of system clock
I20260620 18:42:16.305766 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:35097
--webserver_interface=127.5.225.190
--webserver_port=0
--builtin_ntp_servers=127.5.225.148:38427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:35097
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44607/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980894474392-6022-0/testchainca.cert with env {}
W20260620 18:42:16.624383 11173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:16.624670 11173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:16.624758 11173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:16.631785 11173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:42:16.631965 11173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:16.632033 11173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:16.632097 11173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:42:16.632143 11173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 18:42:16.632200 11173 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 18:42:16.632238 11173 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44607/jwks.json
I20260620 18:42:16.641876 11173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:38427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:35097
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:35097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44607/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980894474392-6022-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:16.643304 11173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:16.644575 11173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:16.656463 11178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:16.656414 11179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:16.656494 11181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:16.657398 11173 server_base.cc:1034] running on GCE node
I20260620 18:42:16.658296 11173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:16.659725 11173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:16.660954 11173 hybrid_clock.cc:648] HybridClock initialized: now 1781980936660911 us; error 58 us; skew 500 ppm
I20260620 18:42:16.661305 11173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:42:16.663568 11173 webserver.cc:460] Webserver started at http://127.5.225.190:41195/ using document root <none> and password file <none>
I20260620 18:42:16.664250 11173 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:16.664383 11173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:16.664688 11173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:16.667567 11173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "ec2ad0c4313c48c5a1707caaa5972d63"
format_stamp: "Formatted at 2026-06-20 18:42:16 on dist-test-slave-k6lg"
server_key: "909fa09f20efe7b23c8ccb6840594b2c"
server_key_iv: "9cef2adcf90bee53795cb4fbcf2f59d2"
server_key_version: "encryptionkey@0"
I20260620 18:42:16.668349 11173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "ec2ad0c4313c48c5a1707caaa5972d63"
format_stamp: "Formatted at 2026-06-20 18:42:16 on dist-test-slave-k6lg"
server_key: "909fa09f20efe7b23c8ccb6840594b2c"
server_key_iv: "9cef2adcf90bee53795cb4fbcf2f59d2"
server_key_version: "encryptionkey@0"
I20260620 18:42:16.673174 11173 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 18:42:16.676450 11187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:16.677714 11173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:42:16.677898 11173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "ec2ad0c4313c48c5a1707caaa5972d63"
format_stamp: "Formatted at 2026-06-20 18:42:16 on dist-test-slave-k6lg"
server_key: "909fa09f20efe7b23c8ccb6840594b2c"
server_key_iv: "9cef2adcf90bee53795cb4fbcf2f59d2"
server_key_version: "encryptionkey@0"
I20260620 18:42:16.678107 11173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:16.691128 11173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:16.691901 11173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:16.692211 11173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:16.710595 11173 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:35097
I20260620 18:42:16.710621 11238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:35097 every 8 connection(s)
I20260620 18:42:16.711951 11173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:42:16.713670 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 11173
I20260620 18:42:16.713891 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:42:16.714560 6022 external_mini_cluster.cc:1428] Setting key bab58ab50ac5cd9816a6e1426a736106
I20260620 18:42:16.716101 11239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:16.727380 11239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec2ad0c4313c48c5a1707caaa5972d63: Bootstrap starting.
I20260620 18:42:16.727396 11242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:16.716404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50846 (local address 127.5.225.190:35097)
0620 18:42:16.717309 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:16.717321 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:16.717359 (+ 38us) server_negotiation.cc:407] Connection header received
0620 18:42:16.718969 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:16.718989 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:16.719432 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:16.719839 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:16.720421 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:16.721738 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:16.723137 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:16.723775 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:16.723983 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:16.725190 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:16.725201 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:16.725214 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:42:16.725771 (+ 557us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:16.725837 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:42:16.725858 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:16.725983 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:16.726218 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 18:42:16.726505 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":301,"threads_started":1}
I20260620 18:42:16.730561 11239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec2ad0c4313c48c5a1707caaa5972d63: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:16.731096 6022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:42:16.731642 11239 log.cc:824] T 00000000000000000000000000000000 P ec2ad0c4313c48c5a1707caaa5972d63: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:16.734424 11239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec2ad0c4313c48c5a1707caaa5972d63: No bootstrap required, opened a new log
I20260620 18:42:16.741855 11239 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec2ad0c4313c48c5a1707caaa5972d63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec2ad0c4313c48c5a1707caaa5972d63" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 35097 } }
I20260620 18:42:16.742276 11239 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec2ad0c4313c48c5a1707caaa5972d63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:16.742394 11239 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec2ad0c4313c48c5a1707caaa5972d63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec2ad0c4313c48c5a1707caaa5972d63, State: Initialized, Role: FOLLOWER
I20260620 18:42:16.743000 11239 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec2ad0c4313c48c5a1707caaa5972d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec2ad0c4313c48c5a1707caaa5972d63" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 35097 } }
I20260620 18:42:16.743233 11239 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec2ad0c4313c48c5a1707caaa5972d63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:42:16.743353 11239 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec2ad0c4313c48c5a1707caaa5972d63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:42:16.743526 11239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec2ad0c4313c48c5a1707caaa5972d63 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:16.745671 11239 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec2ad0c4313c48c5a1707caaa5972d63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec2ad0c4313c48c5a1707caaa5972d63" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 35097 } }
I20260620 18:42:16.746330 11239 leader_election.cc:302] T 00000000000000000000000000000000 P ec2ad0c4313c48c5a1707caaa5972d63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec2ad0c4313c48c5a1707caaa5972d63; no voters:
I20260620 18:42:16.746938 11239 leader_election.cc:288] T 00000000000000000000000000000000 P ec2ad0c4313c48c5a1707caaa5972d63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:42:16.747098 11244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec2ad0c4313c48c5a1707caaa5972d63 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:42:16.747815 11244 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec2ad0c4313c48c5a1707caaa5972d63 [term 1 LEADER]: Becoming Leader. State: Replica: ec2ad0c4313c48c5a1707caaa5972d63, State: Running, Role: LEADER
I20260620 18:42:16.748334 11244 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec2ad0c4313c48c5a1707caaa5972d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec2ad0c4313c48c5a1707caaa5972d63" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 35097 } }
I20260620 18:42:16.749058 11239 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec2ad0c4313c48c5a1707caaa5972d63 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:42:16.751147 11246 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec2ad0c4313c48c5a1707caaa5972d63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec2ad0c4313c48c5a1707caaa5972d63. Latest consensus state: current_term: 1 leader_uuid: "ec2ad0c4313c48c5a1707caaa5972d63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec2ad0c4313c48c5a1707caaa5972d63" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 35097 } } }
I20260620 18:42:16.751626 11246 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec2ad0c4313c48c5a1707caaa5972d63 [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:16.751982 11245 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec2ad0c4313c48c5a1707caaa5972d63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec2ad0c4313c48c5a1707caaa5972d63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec2ad0c4313c48c5a1707caaa5972d63" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 35097 } } }
I20260620 18:42:16.752313 11245 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec2ad0c4313c48c5a1707caaa5972d63 [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:16.754027 11252 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:42:16.758024 11252 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:42:16.766860 11252 catalog_manager.cc:1347] Generated new cluster ID: 26b8e04decb84c99a6601edbf8ba1225
I20260620 18:42:16.766963 11252 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:42:16.777768 11252 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:42:16.779510 11252 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:42:16.788017 11252 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec2ad0c4313c48c5a1707caaa5972d63: Generated new TSK 0
I20260620 18:42:16.788938 11252 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 18:42:16.828569 11269 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 18:42:16.828766 11269 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 18:42:16.829699 11242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:16.808586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50854 (local address 127.5.225.190:35097)
0620 18:42:16.808838 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:16.808845 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:16.809318 (+ 473us) server_negotiation.cc:407] Connection header received
0620 18:42:16.809750 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:16.809755 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:16.809841 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:16.809971 (+ 130us) server_negotiation.cc:226] Negotiated authn=JWT
0620 18:42:16.810680 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:16.811397 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:16.813406 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:16.813821 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:16.813914 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:16.814566 (+ 652us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 18:42:16.829166 (+ 14600us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44607/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 18:42:16.829443 (+ 277us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50854: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44607/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 18:42:16.829836 11242 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50854: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44607/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 18:42:16.831632 11268 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:16.808462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:35097 (local address 127.0.0.1:50854)
0620 18:42:16.809178 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:16.809199 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:16.809406 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:16.810032 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:16.810045 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:16.810058 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0620 18:42:16.810495 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:16.810508 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:16.811533 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:16.811539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:16.813256 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:16.813270 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:16.813980 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:16.813987 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:16.814207 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:16.814287 (+ 80us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 18:42:16.829485 (+ 15198us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44607/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 18:42:16.829849 (+ 364us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.225.190:35097: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44607/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":344,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.5.225.190:35097: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44607/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 18:42:16.833067 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 11173
I20260620 18:42:16.834201 11234 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:16.924846 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 11173
2026-06-20T18:42:16Z chronyd exiting
I20260620 18:42:16.974619 6022 test_util.cc:175] -----------------------------------------------
I20260620 18:42:16.974789 6022 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980894474392-6022-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (702 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 18:42:16.978546 6022 mini_oidc.cc:152] Starting JWKS server
I20260620 18:42:16.980372 6022 webserver.cc:460] Webserver started at https://127.0.0.1:44655/ using document root <none> and password file <none>
I20260620 18:42:16.980644 6022 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 18:42:16.981376 6022 webserver.cc:460] Webserver started at http://127.0.0.1:33815/ using document root <none> and password file <none>
2026-06-20T18:42:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:42:16Z Disabled control of system clock
I20260620 18:42:17.007663 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:44695
--webserver_interface=127.5.225.190
--webserver_port=0
--builtin_ntp_servers=127.5.225.148:43817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:44695
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44655/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980894474392-6022-0/testchainca.cert with env {}
W20260620 18:42:17.327313 11276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:17.327646 11276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:17.327741 11276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:17.335276 11276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:42:17.335400 11276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:17.335438 11276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:17.335489 11276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:42:17.335543 11276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 18:42:17.335599 11276 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 18:42:17.335639 11276 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44655/jwks.json
I20260620 18:42:17.345851 11276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:43817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:44695
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:44695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44655/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980894474392-6022-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:17.347262 11276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:17.348555 11276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:17.360718 11284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:17.360791 11282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:17.360831 11281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:17.361483 11276 server_base.cc:1034] running on GCE node
I20260620 18:42:17.362406 11276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:17.363888 11276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:17.365141 11276 hybrid_clock.cc:648] HybridClock initialized: now 1781980937365082 us; error 45 us; skew 500 ppm
I20260620 18:42:17.365512 11276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:42:17.371945 11276 webserver.cc:460] Webserver started at http://127.5.225.190:35115/ using document root <none> and password file <none>
I20260620 18:42:17.372713 11276 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:17.372859 11276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:17.373199 11276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:17.376349 11276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "9deb29eb75c64b95bf0e0ab80eaefae9"
format_stamp: "Formatted at 2026-06-20 18:42:17 on dist-test-slave-k6lg"
server_key: "1eaea9916714813bfde2f51b68d6b490"
server_key_iv: "6fe50d449af4843331824edbc9e8f7ac"
server_key_version: "encryptionkey@0"
I20260620 18:42:17.377518 11276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "9deb29eb75c64b95bf0e0ab80eaefae9"
format_stamp: "Formatted at 2026-06-20 18:42:17 on dist-test-slave-k6lg"
server_key: "1eaea9916714813bfde2f51b68d6b490"
server_key_iv: "6fe50d449af4843331824edbc9e8f7ac"
server_key_version: "encryptionkey@0"
I20260620 18:42:17.383015 11276 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260620 18:42:17.386868 11290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:17.388469 11276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:42:17.388680 11276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "9deb29eb75c64b95bf0e0ab80eaefae9"
format_stamp: "Formatted at 2026-06-20 18:42:17 on dist-test-slave-k6lg"
server_key: "1eaea9916714813bfde2f51b68d6b490"
server_key_iv: "6fe50d449af4843331824edbc9e8f7ac"
server_key_version: "encryptionkey@0"
I20260620 18:42:17.388911 11276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:17.416167 11276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:17.417073 11276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:17.417387 11276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:17.436223 11276 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:44695
I20260620 18:42:17.436241 11341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:44695 every 8 connection(s)
I20260620 18:42:17.437656 11276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:42:17.441488 11342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:17.446522 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 11276
I20260620 18:42:17.446708 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:42:17.447369 6022 external_mini_cluster.cc:1428] Setting key 348483bb4d3eab11d7c8df3142fc9eba
I20260620 18:42:17.452790 11342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9deb29eb75c64b95bf0e0ab80eaefae9: Bootstrap starting.
I20260620 18:42:17.456122 11342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9deb29eb75c64b95bf0e0ab80eaefae9: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:17.457140 11342 log.cc:824] T 00000000000000000000000000000000 P 9deb29eb75c64b95bf0e0ab80eaefae9: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:17.460273 11342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9deb29eb75c64b95bf0e0ab80eaefae9: No bootstrap required, opened a new log
I20260620 18:42:17.462368 11345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:17.449423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56208 (local address 127.5.225.190:44695)
0620 18:42:17.450680 (+ 1257us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:17.450692 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:17.450745 (+ 53us) server_negotiation.cc:407] Connection header received
0620 18:42:17.452604 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:17.452627 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:17.453207 (+ 580us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:17.453684 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:17.454318 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.455852 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.457355 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.458206 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.458453 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:17.459699 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:17.459707 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:17.459723 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:42:17.460301 (+ 578us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:17.460413 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:42:17.460447 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:17.460583 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:17.460871 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 18:42:17.461245 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":431,"threads_started":1}
I20260620 18:42:17.466897 6022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:42:17.469023 11342 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9deb29eb75c64b95bf0e0ab80eaefae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9deb29eb75c64b95bf0e0ab80eaefae9" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 44695 } }
I20260620 18:42:17.469552 11342 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9deb29eb75c64b95bf0e0ab80eaefae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:17.469683 11342 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9deb29eb75c64b95bf0e0ab80eaefae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9deb29eb75c64b95bf0e0ab80eaefae9, State: Initialized, Role: FOLLOWER
I20260620 18:42:17.470465 11342 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9deb29eb75c64b95bf0e0ab80eaefae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9deb29eb75c64b95bf0e0ab80eaefae9" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 44695 } }
I20260620 18:42:17.470748 11342 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9deb29eb75c64b95bf0e0ab80eaefae9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:42:17.470899 11342 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9deb29eb75c64b95bf0e0ab80eaefae9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:42:17.471091 11342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9deb29eb75c64b95bf0e0ab80eaefae9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:17.473328 11342 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9deb29eb75c64b95bf0e0ab80eaefae9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9deb29eb75c64b95bf0e0ab80eaefae9" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 44695 } }
I20260620 18:42:17.473901 11342 leader_election.cc:302] T 00000000000000000000000000000000 P 9deb29eb75c64b95bf0e0ab80eaefae9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9deb29eb75c64b95bf0e0ab80eaefae9; no voters:
I20260620 18:42:17.474574 11342 leader_election.cc:288] T 00000000000000000000000000000000 P 9deb29eb75c64b95bf0e0ab80eaefae9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:42:17.474784 11347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9deb29eb75c64b95bf0e0ab80eaefae9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:42:17.475562 11347 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9deb29eb75c64b95bf0e0ab80eaefae9 [term 1 LEADER]: Becoming Leader. State: Replica: 9deb29eb75c64b95bf0e0ab80eaefae9, State: Running, Role: LEADER
I20260620 18:42:17.476178 11347 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9deb29eb75c64b95bf0e0ab80eaefae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9deb29eb75c64b95bf0e0ab80eaefae9" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 44695 } }
I20260620 18:42:17.476841 11342 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9deb29eb75c64b95bf0e0ab80eaefae9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:42:17.479542 11348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9deb29eb75c64b95bf0e0ab80eaefae9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9deb29eb75c64b95bf0e0ab80eaefae9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9deb29eb75c64b95bf0e0ab80eaefae9" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 44695 } } }
I20260620 18:42:17.479449 11349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9deb29eb75c64b95bf0e0ab80eaefae9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9deb29eb75c64b95bf0e0ab80eaefae9. Latest consensus state: current_term: 1 leader_uuid: "9deb29eb75c64b95bf0e0ab80eaefae9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9deb29eb75c64b95bf0e0ab80eaefae9" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 44695 } } }
I20260620 18:42:17.479846 11349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9deb29eb75c64b95bf0e0ab80eaefae9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:17.479846 11348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9deb29eb75c64b95bf0e0ab80eaefae9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:17.482296 11355 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:42:17.487749 11355 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:42:17.497376 11355 catalog_manager.cc:1347] Generated new cluster ID: 355dd255639b4016bbe80058542e1f13
I20260620 18:42:17.497514 11355 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:42:17.518289 11355 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:42:17.519634 11355 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:42:17.528218 11355 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9deb29eb75c64b95bf0e0ab80eaefae9: Generated new TSK 0
I20260620 18:42:17.529099 11355 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 18:42:17.568900 11372 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 18:42:17.569041 11372 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 18:42:17.570349 11345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:17.549878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56222 (local address 127.5.225.190:44695)
0620 18:42:17.550084 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:17.550091 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:17.550542 (+ 451us) server_negotiation.cc:407] Connection header received
0620 18:42:17.550763 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:17.550770 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:17.550865 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:17.551035 (+ 170us) server_negotiation.cc:226] Negotiated authn=JWT
0620 18:42:17.551485 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.552632 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.554018 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.554794 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.554931 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:17.555529 (+ 598us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 18:42:17.569653 (+ 14124us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44655/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 18:42:17.570031 (+ 378us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56222: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44655/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 18:42:17.570485 11345 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56222: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44655/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 18:42:17.570567 11371 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:17.549749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:44695 (local address 127.0.0.1:56222)
0620 18:42:17.550439 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:17.550455 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:17.550580 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:17.551100 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:17.551103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:17.551113 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0620 18:42:17.551343 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:17.551350 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.552800 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.552804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:17.553875 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:17.553886 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.555003 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.555007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:17.555190 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:17.555239 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 18:42:17.569927 (+ 14688us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44655/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 18:42:17.570154 (+ 227us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.225.190:44695: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44655/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":434,"threads_started":1}
I20260620 18:42:17.572829 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 11276
I20260620 18:42:17.573999 11337 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:17.668859 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 11276
2026-06-20T18:42:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (745 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 18:42:17.724722 6022 mini_oidc.cc:152] Starting JWKS server
I20260620 18:42:17.726758 6022 webserver.cc:460] Webserver started at https://127.0.0.1:39243/ using document root <none> and password file <none>
I20260620 18:42:17.727046 6022 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 18:42:17.727730 6022 webserver.cc:460] Webserver started at http://127.0.0.1:34919/ using document root <none> and password file <none>
2026-06-20T18:42:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:42:17Z Disabled control of system clock
I20260620 18:42:17.755213 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:36513
--webserver_interface=127.5.225.190
--webserver_port=0
--builtin_ntp_servers=127.5.225.148:36343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:36513
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39243/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 18:42:18.077843 11379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:18.078235 11379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:18.078397 11379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:18.085902 11379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:42:18.086054 11379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:18.086148 11379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:18.086300 11379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:42:18.086390 11379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 18:42:18.086478 11379 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 18:42:18.086560 11379 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39243/jwks.json
I20260620 18:42:18.096001 11379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:36343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:36513
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:36513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39243/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:18.097416 11379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:18.099376 11379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:18.111431 11384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:18.111486 11385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:18.111510 11387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:18.112409 11379 server_base.cc:1034] running on GCE node
I20260620 18:42:18.113202 11379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:18.114682 11379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:18.115882 11379 hybrid_clock.cc:648] HybridClock initialized: now 1781980938115848 us; error 30 us; skew 500 ppm
I20260620 18:42:18.116247 11379 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:42:18.118585 11379 webserver.cc:460] Webserver started at http://127.5.225.190:38185/ using document root <none> and password file <none>
I20260620 18:42:18.119270 11379 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:18.119381 11379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:18.119714 11379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:18.122642 11379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "8fa9ad8413024ae1a925d8d812d8c1ed"
format_stamp: "Formatted at 2026-06-20 18:42:18 on dist-test-slave-k6lg"
server_key: "ce335ae24fb74a91ea40093cffebc3b8"
server_key_iv: "e1c73d542077f0e5302b24b9184e61a0"
server_key_version: "encryptionkey@0"
I20260620 18:42:18.123428 11379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "8fa9ad8413024ae1a925d8d812d8c1ed"
format_stamp: "Formatted at 2026-06-20 18:42:18 on dist-test-slave-k6lg"
server_key: "ce335ae24fb74a91ea40093cffebc3b8"
server_key_iv: "e1c73d542077f0e5302b24b9184e61a0"
server_key_version: "encryptionkey@0"
I20260620 18:42:18.128472 11379 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:42:18.131814 11393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:18.133177 11379 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:42:18.133368 11379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "8fa9ad8413024ae1a925d8d812d8c1ed"
format_stamp: "Formatted at 2026-06-20 18:42:18 on dist-test-slave-k6lg"
server_key: "ce335ae24fb74a91ea40093cffebc3b8"
server_key_iv: "e1c73d542077f0e5302b24b9184e61a0"
server_key_version: "encryptionkey@0"
I20260620 18:42:18.133589 11379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:18.148938 11379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:18.149775 11379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:18.150087 11379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:18.168998 11379 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:36513
I20260620 18:42:18.169015 11444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:36513 every 8 connection(s)
I20260620 18:42:18.170406 11379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:42:18.174024 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 11379
I20260620 18:42:18.174260 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:42:18.174487 11445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:18.175071 6022 external_mini_cluster.cc:1428] Setting key e41970c8659d60bbc06a2316d5c1e992
I20260620 18:42:18.187685 11445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fa9ad8413024ae1a925d8d812d8c1ed: Bootstrap starting.
I20260620 18:42:18.189182 11448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:18.176953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50686 (local address 127.5.225.190:36513)
0620 18:42:18.178091 (+ 1138us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:18.178105 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:18.178155 (+ 50us) server_negotiation.cc:407] Connection header received
0620 18:42:18.180410 (+ 2255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:18.180433 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:18.180911 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:18.181239 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:18.181750 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:18.183051 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:18.184892 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:18.185572 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:18.185702 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:18.186912 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:18.186920 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:18.186933 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:42:18.187449 (+ 516us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:18.187520 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:42:18.187544 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:18.187627 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:18.187915 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 18:42:18.188249 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":519,"threads_started":1}
I20260620 18:42:18.190824 11445 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fa9ad8413024ae1a925d8d812d8c1ed: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:18.191955 11445 log.cc:824] T 00000000000000000000000000000000 P 8fa9ad8413024ae1a925d8d812d8c1ed: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:18.193008 6022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:42:18.195158 11445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fa9ad8413024ae1a925d8d812d8c1ed: No bootstrap required, opened a new log
I20260620 18:42:18.202867 11445 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fa9ad8413024ae1a925d8d812d8c1ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fa9ad8413024ae1a925d8d812d8c1ed" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36513 } }
I20260620 18:42:18.203193 11445 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8fa9ad8413024ae1a925d8d812d8c1ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:18.203264 11445 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fa9ad8413024ae1a925d8d812d8c1ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fa9ad8413024ae1a925d8d812d8c1ed, State: Initialized, Role: FOLLOWER
I20260620 18:42:18.203718 11445 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fa9ad8413024ae1a925d8d812d8c1ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fa9ad8413024ae1a925d8d812d8c1ed" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36513 } }
I20260620 18:42:18.203907 11445 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8fa9ad8413024ae1a925d8d812d8c1ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:42:18.203997 11445 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fa9ad8413024ae1a925d8d812d8c1ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:42:18.204119 11445 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fa9ad8413024ae1a925d8d812d8c1ed [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:18.206096 11445 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fa9ad8413024ae1a925d8d812d8c1ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fa9ad8413024ae1a925d8d812d8c1ed" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36513 } }
I20260620 18:42:18.206660 11445 leader_election.cc:302] T 00000000000000000000000000000000 P 8fa9ad8413024ae1a925d8d812d8c1ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8fa9ad8413024ae1a925d8d812d8c1ed; no voters:
I20260620 18:42:18.207195 11445 leader_election.cc:288] T 00000000000000000000000000000000 P 8fa9ad8413024ae1a925d8d812d8c1ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:42:18.207345 11450 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8fa9ad8413024ae1a925d8d812d8c1ed [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:42:18.208036 11450 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8fa9ad8413024ae1a925d8d812d8c1ed [term 1 LEADER]: Becoming Leader. State: Replica: 8fa9ad8413024ae1a925d8d812d8c1ed, State: Running, Role: LEADER
I20260620 18:42:18.208527 11450 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8fa9ad8413024ae1a925d8d812d8c1ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fa9ad8413024ae1a925d8d812d8c1ed" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36513 } }
I20260620 18:42:18.209203 11445 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fa9ad8413024ae1a925d8d812d8c1ed [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:42:18.211576 11452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fa9ad8413024ae1a925d8d812d8c1ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fa9ad8413024ae1a925d8d812d8c1ed. Latest consensus state: current_term: 1 leader_uuid: "8fa9ad8413024ae1a925d8d812d8c1ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fa9ad8413024ae1a925d8d812d8c1ed" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36513 } } }
I20260620 18:42:18.211971 11452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fa9ad8413024ae1a925d8d812d8c1ed [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:18.212404 11451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fa9ad8413024ae1a925d8d812d8c1ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8fa9ad8413024ae1a925d8d812d8c1ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fa9ad8413024ae1a925d8d812d8c1ed" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36513 } } }
I20260620 18:42:18.212728 11451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fa9ad8413024ae1a925d8d812d8c1ed [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:18.213557 11459 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:42:18.218730 11459 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:42:18.228551 11459 catalog_manager.cc:1347] Generated new cluster ID: 245f42853f5e4e6c8685d80545c3a827
I20260620 18:42:18.228665 11459 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:42:18.241180 11459 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:42:18.242556 11459 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:42:18.252115 11459 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8fa9ad8413024ae1a925d8d812d8c1ed: Generated new TSK 0
I20260620 18:42:18.253013 11459 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:42:18.271922 11448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:18.264432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50702 (local address 127.5.225.190:36513)
0620 18:42:18.264635 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:18.264642 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:18.264969 (+ 327us) server_negotiation.cc:407] Connection header received
0620 18:42:18.265230 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:18.265234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:18.265306 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:18.265452 (+ 146us) server_negotiation.cc:226] Negotiated authn=JWT
0620 18:42:18.265898 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:18.266673 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:18.268010 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:18.268437 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:18.268571 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:18.269042 (+ 471us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 18:42:18.271334 (+ 2292us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39243/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 18:42:18.271645 (+ 311us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50702: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39243/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 18:42:18.272035 11448 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50702: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39243/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 18:42:18.274263 11474 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:18.264298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36513 (local address 127.0.0.1:50702)
0620 18:42:18.264859 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:18.264888 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:18.265040 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:18.265467 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:18.265472 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:18.265484 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0620 18:42:18.265754 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:18.265763 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:18.266819 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:18.266825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:18.267865 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:18.267876 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:18.268621 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:18.268628 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:18.268773 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:18.268803 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 18:42:18.271552 (+ 2749us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39243/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 18:42:18.273753 (+ 2201us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.225.190:36513: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39243/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":1884,"thread_start_us":362,"threads_started":1}
I20260620 18:42:18.275291 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 11379
I20260620 18:42:18.276345 11440 generic_service.cc:196] Checking for leaks (request via RPC)
E20260620 18:42:18.284344 11475 webserver.cc:533] Webserver: sslize failed:
E20260620 18:42:18.284474 11475 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 18:42:18.365765 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 11379
2026-06-20T18:42:18Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (692 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T18:42:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:42:18Z Disabled control of system clock
I20260620 18:42:18.443734 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:44083
--webserver_interface=127.5.225.190
--webserver_port=0
--builtin_ntp_servers=127.5.225.148:38199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:44083
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980894474392-6022-0/testchainca.cert with env {}
W20260620 18:42:18.760421 11480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:18.760720 11480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:18.760807 11480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:18.768055 11480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:42:18.768167 11480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:18.768204 11480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:18.768270 11480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:42:18.768313 11480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 18:42:18.768373 11480 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 18:42:18.768412 11480 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 18:42:18.778059 11480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:38199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:44083
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:44083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980894474392-6022-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:18.779470 11480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:18.780764 11480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:18.792661 11486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:18.792686 11488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:18.792600 11485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:18.793669 11480 server_base.cc:1034] running on GCE node
I20260620 18:42:18.794450 11480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:18.795846 11480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:18.797052 11480 hybrid_clock.cc:648] HybridClock initialized: now 1781980938797006 us; error 43 us; skew 500 ppm
I20260620 18:42:18.797415 11480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:42:18.799860 11480 webserver.cc:460] Webserver started at http://127.5.225.190:36363/ using document root <none> and password file <none>
I20260620 18:42:18.800560 11480 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:18.800648 11480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:18.800992 11480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:18.803881 11480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "33ee536ea9ba45f6bfbafa5e5758de6a"
format_stamp: "Formatted at 2026-06-20 18:42:18 on dist-test-slave-k6lg"
server_key: "b8ce57b838ec552a6b4c56f942d72f4f"
server_key_iv: "8d56d98fffaf7ec4cba73cf16641fdc7"
server_key_version: "encryptionkey@0"
I20260620 18:42:18.804688 11480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "33ee536ea9ba45f6bfbafa5e5758de6a"
format_stamp: "Formatted at 2026-06-20 18:42:18 on dist-test-slave-k6lg"
server_key: "b8ce57b838ec552a6b4c56f942d72f4f"
server_key_iv: "8d56d98fffaf7ec4cba73cf16641fdc7"
server_key_version: "encryptionkey@0"
I20260620 18:42:18.809744 11480 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260620 18:42:18.813184 11494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:18.814682 11480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:42:18.814870 11480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "33ee536ea9ba45f6bfbafa5e5758de6a"
format_stamp: "Formatted at 2026-06-20 18:42:18 on dist-test-slave-k6lg"
server_key: "b8ce57b838ec552a6b4c56f942d72f4f"
server_key_iv: "8d56d98fffaf7ec4cba73cf16641fdc7"
server_key_version: "encryptionkey@0"
I20260620 18:42:18.815081 11480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:18.844686 11480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:18.845551 11480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:18.845849 11480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:18.864569 11480 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:44083
I20260620 18:42:18.864589 11545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:44083 every 8 connection(s)
I20260620 18:42:18.865949 11480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:42:18.869658 11546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:18.871739 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 11480
I20260620 18:42:18.871929 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:42:18.872768 6022 external_mini_cluster.cc:1428] Setting key 92e47d9212c67f0041667cd368fd0565
I20260620 18:42:18.881227 11546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33ee536ea9ba45f6bfbafa5e5758de6a: Bootstrap starting.
I20260620 18:42:18.884271 11546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33ee536ea9ba45f6bfbafa5e5758de6a: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:18.885376 11546 log.cc:824] T 00000000000000000000000000000000 P 33ee536ea9ba45f6bfbafa5e5758de6a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:18.886284 11549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:18.874824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48444 (local address 127.5.225.190:44083)
0620 18:42:18.875774 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:18.875784 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:18.875830 (+ 46us) server_negotiation.cc:407] Connection header received
0620 18:42:18.877581 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:18.877602 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:18.878054 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:18.878511 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:18.879073 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:18.880300 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:18.881638 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:18.882379 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:18.882604 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:18.883762 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:18.883770 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:18.883784 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:42:18.884301 (+ 517us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:18.884419 (+ 118us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:42:18.884452 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:18.884589 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:18.884828 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 18:42:18.885133 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":316,"threads_started":1}
I20260620 18:42:18.888454 11546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33ee536ea9ba45f6bfbafa5e5758de6a: No bootstrap required, opened a new log
I20260620 18:42:18.890010 6022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:42:18.896376 11546 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33ee536ea9ba45f6bfbafa5e5758de6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33ee536ea9ba45f6bfbafa5e5758de6a" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 44083 } }
I20260620 18:42:18.896714 11546 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33ee536ea9ba45f6bfbafa5e5758de6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:18.896828 11546 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33ee536ea9ba45f6bfbafa5e5758de6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33ee536ea9ba45f6bfbafa5e5758de6a, State: Initialized, Role: FOLLOWER
I20260620 18:42:18.897426 11546 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33ee536ea9ba45f6bfbafa5e5758de6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33ee536ea9ba45f6bfbafa5e5758de6a" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 44083 } }
I20260620 18:42:18.897665 11546 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33ee536ea9ba45f6bfbafa5e5758de6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:42:18.897792 11546 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33ee536ea9ba45f6bfbafa5e5758de6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:42:18.897958 11546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33ee536ea9ba45f6bfbafa5e5758de6a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:18.900269 11546 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33ee536ea9ba45f6bfbafa5e5758de6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33ee536ea9ba45f6bfbafa5e5758de6a" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 44083 } }
I20260620 18:42:18.900767 11546 leader_election.cc:302] T 00000000000000000000000000000000 P 33ee536ea9ba45f6bfbafa5e5758de6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33ee536ea9ba45f6bfbafa5e5758de6a; no voters:
I20260620 18:42:18.901336 11546 leader_election.cc:288] T 00000000000000000000000000000000 P 33ee536ea9ba45f6bfbafa5e5758de6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:42:18.901526 11551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33ee536ea9ba45f6bfbafa5e5758de6a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:42:18.902248 11551 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33ee536ea9ba45f6bfbafa5e5758de6a [term 1 LEADER]: Becoming Leader. State: Replica: 33ee536ea9ba45f6bfbafa5e5758de6a, State: Running, Role: LEADER
I20260620 18:42:18.902788 11551 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33ee536ea9ba45f6bfbafa5e5758de6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33ee536ea9ba45f6bfbafa5e5758de6a" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 44083 } }
I20260620 18:42:18.903422 11546 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33ee536ea9ba45f6bfbafa5e5758de6a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:42:18.905678 11552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33ee536ea9ba45f6bfbafa5e5758de6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33ee536ea9ba45f6bfbafa5e5758de6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33ee536ea9ba45f6bfbafa5e5758de6a" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 44083 } } }
I20260620 18:42:18.905704 11553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33ee536ea9ba45f6bfbafa5e5758de6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33ee536ea9ba45f6bfbafa5e5758de6a. Latest consensus state: current_term: 1 leader_uuid: "33ee536ea9ba45f6bfbafa5e5758de6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33ee536ea9ba45f6bfbafa5e5758de6a" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 44083 } } }
I20260620 18:42:18.906106 11552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33ee536ea9ba45f6bfbafa5e5758de6a [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:18.906106 11553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33ee536ea9ba45f6bfbafa5e5758de6a [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:18.907867 11556 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:42:18.913394 11556 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:42:18.922039 11556 catalog_manager.cc:1347] Generated new cluster ID: eab06e2f7c3246458609b7d0c3b615f7
I20260620 18:42:18.922154 11556 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:42:18.945070 11556 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:42:18.946955 11556 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:42:18.961587 11556 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33ee536ea9ba45f6bfbafa5e5758de6a: Generated new TSK 0
I20260620 18:42:18.962646 11556 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:42:19.039235 11549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:18.977795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48454 (local address 127.5.225.190:44083)
0620 18:42:18.977980 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:18.977986 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:18.978477 (+ 491us) server_negotiation.cc:407] Connection header received
0620 18:42:18.978659 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:18.978664 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:18.978758 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:18.978908 (+ 150us) server_negotiation.cc:226] Negotiated authn=JWT
0620 18:42:18.979337 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:18.980446 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:18.981773 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:18.982447 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:18.982579 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:18.983000 (+ 421us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 18:42:19.038438 (+ 55438us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 18:42:19.038872 (+ 434us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48454: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 18:42:19.039417 11575 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:18.977763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:44083 (local address 127.0.0.1:48454)
0620 18:42:18.978378 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:18.978395 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:18.978506 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:18.978933 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:18.978936 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:18.978944 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0620 18:42:18.979195 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:18.979202 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:18.980588 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:18.980592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:18.981639 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:18.981648 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:18.982607 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:18.982612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:18.982727 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:18.982756 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 18:42:19.038738 (+ 55982us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 18:42:19.039022 (+ 284us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.225.190:44083: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":321,"threads_started":1}
W20260620 18:42:19.039413 11549 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48454: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260620 18:42:19.040740 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 11480
I20260620 18:42:19.042068 11541 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:19.139389 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 11480
2026-06-20T18:42:19Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (756 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (404 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (402 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:42:20 dist-test-slave-k6lg krb5kdc[11590](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:42:20 dist-test-slave-k6lg krb5kdc[11590](info): set up 2 sockets
Jun 20 18:42:20 dist-test-slave-k6lg krb5kdc[11590](info): commencing operation
krb5kdc: starting...
W20260620 18:42:21.030937 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:42:21 dist-test-slave-k6lg krb5kdc[11590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:42:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:42:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
I20260620 18:42:21.233708 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:42269
--webserver_interface=127.5.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:35911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:42269
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:21.553923 11603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:21.554219 11603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:21.554369 11603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:21.561523 11603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:42:21.561631 11603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:21.561729 11603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:21.561825 11603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:42:21.561909 11603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:42:21.571666 11603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:35911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:42269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:42269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:21.572916 11603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:21.574270 11603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:21.585757 11609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:21.585805 11608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:21.585829 11611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:21.586632 11603 server_base.cc:1034] running on GCE node
I20260620 18:42:21.587344 11603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:21.588646 11603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:21.589875 11603 hybrid_clock.cc:648] HybridClock initialized: now 1781980941589825 us; error 56 us; skew 500 ppm
I20260620 18:42:21.590270 11603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:21 dist-test-slave-k6lg krb5kdc[11590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980941, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:21.593354 11603 init.cc:377] Logged in from keytab as kudu/127.5.225.190@KRBTEST.COM (short username kudu)
I20260620 18:42:21.594759 11603 webserver.cc:460] Webserver started at http://127.5.225.190:37665/ using document root <none> and password file <none>
I20260620 18:42:21.595414 11603 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:21.595558 11603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:21.595854 11603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:21.598690 11603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "41d6b9ad3d3c4b19addbf72fc77bc68f"
format_stamp: "Formatted at 2026-06-20 18:42:21 on dist-test-slave-k6lg"
server_key: "f5dd2f1e7c0fd4596f5b4d0bc8d70ad4"
server_key_iv: "32c7ffe599d645270f5b10281ec0a952"
server_key_version: "encryptionkey@0"
I20260620 18:42:21.599438 11603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "41d6b9ad3d3c4b19addbf72fc77bc68f"
format_stamp: "Formatted at 2026-06-20 18:42:21 on dist-test-slave-k6lg"
server_key: "f5dd2f1e7c0fd4596f5b4d0bc8d70ad4"
server_key_iv: "32c7ffe599d645270f5b10281ec0a952"
server_key_version: "encryptionkey@0"
I20260620 18:42:21.604261 11603 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 18:42:21.607441 11618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:21.608654 11603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:42:21.608878 11603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "41d6b9ad3d3c4b19addbf72fc77bc68f"
format_stamp: "Formatted at 2026-06-20 18:42:21 on dist-test-slave-k6lg"
server_key: "f5dd2f1e7c0fd4596f5b4d0bc8d70ad4"
server_key_iv: "32c7ffe599d645270f5b10281ec0a952"
server_key_version: "encryptionkey@0"
I20260620 18:42:21.609118 11603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:21.621474 11603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:21.625212 11603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:21.625509 11603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:21.643329 11603 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:42269
I20260620 18:42:21.643347 11669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:42269 every 8 connection(s)
I20260620 18:42:21.644713 11603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:42:21.648360 11670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:21.651748 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 11603
I20260620 18:42:21.651950 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:42:21.652573 6022 external_mini_cluster.cc:1428] Setting key dff705345625fe7345716721e2fd20fe
I20260620 18:42:21.658847 11670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41d6b9ad3d3c4b19addbf72fc77bc68f: Bootstrap starting.
I20260620 18:42:21.662082 11670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41d6b9ad3d3c4b19addbf72fc77bc68f: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:21.663090 11670 log.cc:824] T 00000000000000000000000000000000 P 41d6b9ad3d3c4b19addbf72fc77bc68f: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:21.665793 11670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41d6b9ad3d3c4b19addbf72fc77bc68f: No bootstrap required, opened a new log
Jun 20 18:42:21 dist-test-slave-k6lg krb5kdc[11590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:42:21.673370 11670 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41d6b9ad3d3c4b19addbf72fc77bc68f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d6b9ad3d3c4b19addbf72fc77bc68f" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 42269 } }
I20260620 18:42:21.673732 11670 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41d6b9ad3d3c4b19addbf72fc77bc68f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:21.673854 11670 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41d6b9ad3d3c4b19addbf72fc77bc68f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41d6b9ad3d3c4b19addbf72fc77bc68f, State: Initialized, Role: FOLLOWER
I20260620 18:42:21.674487 11670 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41d6b9ad3d3c4b19addbf72fc77bc68f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d6b9ad3d3c4b19addbf72fc77bc68f" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 42269 } }
I20260620 18:42:21.674736 11670 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41d6b9ad3d3c4b19addbf72fc77bc68f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:42:21.674881 11670 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41d6b9ad3d3c4b19addbf72fc77bc68f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:42:21.675076 11670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41d6b9ad3d3c4b19addbf72fc77bc68f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:21.675374 11673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:21.654508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46544 (local address 127.5.225.190:42269)
0620 18:42:21.655258 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:21.655270 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:21.655310 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:42:21.657634 (+ 2324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:21.657656 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:21.658038 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:21.658502 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:21.659936 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:21.661422 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:21.662754 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:21.663490 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:21.663742 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:21.667980 (+ 4238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:21.667990 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:21.668003 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:21.668037 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:21.671202 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:21.671933 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:21.671939 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:21.671945 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:21.672106 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:21.672508 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:21.672512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:21.672514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:21.673020 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:21.673313 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:21.673786 (+ 473us) server_negotiation.cc:299] Negotiation successful
0620 18:42:21.674257 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":347,"threads_started":1}
I20260620 18:42:21.677836 11670 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41d6b9ad3d3c4b19addbf72fc77bc68f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d6b9ad3d3c4b19addbf72fc77bc68f" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 42269 } }
I20260620 18:42:21.678467 11670 leader_election.cc:302] T 00000000000000000000000000000000 P 41d6b9ad3d3c4b19addbf72fc77bc68f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41d6b9ad3d3c4b19addbf72fc77bc68f; no voters:
I20260620 18:42:21.679014 11670 leader_election.cc:288] T 00000000000000000000000000000000 P 41d6b9ad3d3c4b19addbf72fc77bc68f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:42:21.679143 11675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41d6b9ad3d3c4b19addbf72fc77bc68f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:42:21.679873 11675 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41d6b9ad3d3c4b19addbf72fc77bc68f [term 1 LEADER]: Becoming Leader. State: Replica: 41d6b9ad3d3c4b19addbf72fc77bc68f, State: Running, Role: LEADER
I20260620 18:42:21.680429 11675 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41d6b9ad3d3c4b19addbf72fc77bc68f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d6b9ad3d3c4b19addbf72fc77bc68f" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 42269 } }
I20260620 18:42:21.680943 11670 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41d6b9ad3d3c4b19addbf72fc77bc68f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:42:21.682928 11676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41d6b9ad3d3c4b19addbf72fc77bc68f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41d6b9ad3d3c4b19addbf72fc77bc68f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d6b9ad3d3c4b19addbf72fc77bc68f" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 42269 } } }
I20260620 18:42:21.683176 11677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41d6b9ad3d3c4b19addbf72fc77bc68f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41d6b9ad3d3c4b19addbf72fc77bc68f. Latest consensus state: current_term: 1 leader_uuid: "41d6b9ad3d3c4b19addbf72fc77bc68f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d6b9ad3d3c4b19addbf72fc77bc68f" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 42269 } } }
I20260620 18:42:21.683365 11676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41d6b9ad3d3c4b19addbf72fc77bc68f [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:21.683420 11677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41d6b9ad3d3c4b19addbf72fc77bc68f [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:21.687165 11680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:42:21.693199 11680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:42:21.701561 11680 catalog_manager.cc:1347] Generated new cluster ID: 905c85c55b394881acd2cb4a3878add0
I20260620 18:42:21.701675 11680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:42:21.712733 11680 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:42:21.714222 11680 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:42:21.726894 11680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41d6b9ad3d3c4b19addbf72fc77bc68f: Generated new TSK 0
I20260620 18:42:21.728060 11680 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab.
Entry for principal kudu/127.5.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab.
I20260620 18:42:21.786600 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.129:0
--local_ip_for_outbound_sockets=127.5.225.129
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:42269
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:35911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:22.108137 11696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:22.108404 11696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:22.108559 11696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:22.116206 11696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:22.116320 11696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:22.116493 11696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.129
I20260620 18:42:22.126362 11696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:35911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:42269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.129
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:22.127734 11696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:22.129112 11696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:22.142491 11701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:22.142447 11702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:22.142527 11704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:22.143143 11696 server_base.cc:1034] running on GCE node
I20260620 18:42:22.143801 11696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:22.144737 11696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:22.145938 11696 hybrid_clock.cc:648] HybridClock initialized: now 1781980942145880 us; error 57 us; skew 500 ppm
I20260620 18:42:22.146322 11696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:22 dist-test-slave-k6lg krb5kdc[11590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980942, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:22.149524 11696 init.cc:377] Logged in from keytab as kudu/127.5.225.129@KRBTEST.COM (short username kudu)
I20260620 18:42:22.151145 11696 webserver.cc:460] Webserver started at http://127.5.225.129:40267/ using document root <none> and password file <none>
I20260620 18:42:22.151858 11696 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:22.152004 11696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:22.152326 11696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:22.155452 11696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-0/data/instance:
uuid: "967a78db21ea492bad2c327ede0a8543"
format_stamp: "Formatted at 2026-06-20 18:42:22 on dist-test-slave-k6lg"
server_key: "bf9ee0a1a8421e2eb140a40c103d0565"
server_key_iv: "31d8f5080985449de86a7ffc15be0823"
server_key_version: "encryptionkey@0"
I20260620 18:42:22.156306 11696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance:
uuid: "967a78db21ea492bad2c327ede0a8543"
format_stamp: "Formatted at 2026-06-20 18:42:22 on dist-test-slave-k6lg"
server_key: "bf9ee0a1a8421e2eb140a40c103d0565"
server_key_iv: "31d8f5080985449de86a7ffc15be0823"
server_key_version: "encryptionkey@0"
I20260620 18:42:22.161943 11696 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 18:42:22.165897 11711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:22.167470 11696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:42:22.167680 11696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-0/wal
uuid: "967a78db21ea492bad2c327ede0a8543"
format_stamp: "Formatted at 2026-06-20 18:42:22 on dist-test-slave-k6lg"
server_key: "bf9ee0a1a8421e2eb140a40c103d0565"
server_key_iv: "31d8f5080985449de86a7ffc15be0823"
server_key_version: "encryptionkey@0"
I20260620 18:42:22.167912 11696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:22.182659 11696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:22.186775 11696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:22.187049 11696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:22.187942 11696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:22.189232 11696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:22.189330 11696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:22.189471 11696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:22.189564 11696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:22.222409 11696 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.129:46301
I20260620 18:42:22.222445 11823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.129:46301 every 8 connection(s)
I20260620 18:42:22.223703 11696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
I20260620 18:42:22.227680 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 11696
I20260620 18:42:22.227880 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance
I20260620 18:42:22.228570 6022 external_mini_cluster.cc:1428] Setting key 95b4ca8b826834049b6a8e263a172f4f
Jun 20 18:42:22 dist-test-slave-k6lg krb5kdc[11590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980942, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:42:22.245115 11826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:22.226324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:43597 (local address 127.5.225.190:42269)
0620 18:42:22.226772 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:22.226776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:22.227708 (+ 932us) server_negotiation.cc:407] Connection header received
0620 18:42:22.229841 (+ 2133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:22.229846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:22.229933 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:22.230069 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:22.231560 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:22.232279 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:22.233833 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:22.234329 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:22.234467 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:22.238920 (+ 4453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:22.238927 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:22.238933 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:22.238973 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:22.241738 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:22.242774 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:22.242778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:22.242780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:22.242894 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:22.243400 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:22.243403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:22.243405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:22.243656 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:22.243764 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:22.244704 (+ 940us) server_negotiation.cc:299] Negotiation successful
0620 18:42:22.244865 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":253,"threads_started":1}
I20260620 18:42:22.246970 11827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:22.226857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:42269 (local address 127.5.225.129:43597)
0620 18:42:22.227548 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:22.227594 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:22.229531 (+ 1937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:22.230335 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:22.230343 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:22.230847 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:22.231398 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:22.231414 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:22.232433 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:22.232438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:22.233691 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:22.233704 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:22.234553 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:22.234562 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:22.234806 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:22.235462 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:22.235497 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:22.238679 (+ 3182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:22.241902 (+ 3223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:22.241914 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:22.241934 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:22.242640 (+ 706us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:22.243007 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:22.243014 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:22.243021 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:22.243288 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:22.243789 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:22.243800 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:22.244299 (+ 499us) client_negotiation.cc:769] Sending connection context
0620 18:42:22.244632 (+ 333us) client_negotiation.cc:240] Negotiation successful
0620 18:42:22.245359 (+ 727us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":297,"threads_started":1}
I20260620 18:42:22.248504 11824 heartbeater.cc:347] Connected to a master server at 127.5.225.190:42269
WARNING: no policy specified for kudu/127.5.225.130@KRBTEST.COM; defaulting to no policy
I20260620 18:42:22.248854 11824 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:22.249621 11824 heartbeater.cc:511] Master 127.5.225.190:42269 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.130@KRBTEST.COM" created.
I20260620 18:42:22.252049 11635 ts_manager.cc:194] Registered new tserver with Master: 967a78db21ea492bad2c327ede0a8543 (127.5.225.129:46301)
I20260620 18:42:22.254278 11635 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.129@KRBTEST.COM'} at 127.5.225.129:43597
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab.
Entry for principal kudu/127.5.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab.
I20260620 18:42:22.278010 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.130:0
--local_ip_for_outbound_sockets=127.5.225.130
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:42269
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:35911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:22.596872 11830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:22.597137 11830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:22.597287 11830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:22.604486 11830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:22.604606 11830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:22.604785 11830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.130
I20260620 18:42:22.614796 11830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:35911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:42269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.130
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:22.616078 11830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:22.617347 11830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:22.630287 11835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:22.630307 11836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:22.630409 11838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:22.631033 11830 server_base.cc:1034] running on GCE node
I20260620 18:42:22.631608 11830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:22.632488 11830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:22.633723 11830 hybrid_clock.cc:648] HybridClock initialized: now 1781980942633674 us; error 61 us; skew 500 ppm
I20260620 18:42:22.634030 11830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:22 dist-test-slave-k6lg krb5kdc[11590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980942, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:22.640022 11830 init.cc:377] Logged in from keytab as kudu/127.5.225.130@KRBTEST.COM (short username kudu)
I20260620 18:42:22.641568 11830 webserver.cc:460] Webserver started at http://127.5.225.130:44343/ using document root <none> and password file <none>
I20260620 18:42:22.642422 11830 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:22.642653 11830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:22.643002 11830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:22.646272 11830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-1/data/instance:
uuid: "a02d68d710914ec3979c778bdb81ada6"
format_stamp: "Formatted at 2026-06-20 18:42:22 on dist-test-slave-k6lg"
server_key: "95e24014c50f01dc0ae0d8a0afc23eba"
server_key_iv: "8d10fc6661906cf99652d3fa7a41b295"
server_key_version: "encryptionkey@0"
I20260620 18:42:22.647148 11830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance:
uuid: "a02d68d710914ec3979c778bdb81ada6"
format_stamp: "Formatted at 2026-06-20 18:42:22 on dist-test-slave-k6lg"
server_key: "95e24014c50f01dc0ae0d8a0afc23eba"
server_key_iv: "8d10fc6661906cf99652d3fa7a41b295"
server_key_version: "encryptionkey@0"
I20260620 18:42:22.652724 11830 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260620 18:42:22.656406 11845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:22.657799 11830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:42:22.657994 11830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-1/wal
uuid: "a02d68d710914ec3979c778bdb81ada6"
format_stamp: "Formatted at 2026-06-20 18:42:22 on dist-test-slave-k6lg"
server_key: "95e24014c50f01dc0ae0d8a0afc23eba"
server_key_iv: "8d10fc6661906cf99652d3fa7a41b295"
server_key_version: "encryptionkey@0"
I20260620 18:42:22.658260 11830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:22.679804 11830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:22.684568 11830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:22.684880 11830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:22.686007 11830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:22.687453 11830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:22.687569 11830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:22.687717 11830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:22.687789 11830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:22.720350 11830 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.130:42515
I20260620 18:42:22.720402 11957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.130:42515 every 8 connection(s)
I20260620 18:42:22.721621 11830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
I20260620 18:42:22.727432 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 11830
I20260620 18:42:22.727619 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance
I20260620 18:42:22.728067 6022 external_mini_cluster.cc:1428] Setting key bfc86a3eef252bf620caf28a85e81490
Jun 20 18:42:22 dist-test-slave-k6lg krb5kdc[11590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980942, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:42:22.743227 11826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:22.724203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:60037 (local address 127.5.225.190:42269)
0620 18:42:22.724373 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:22.724377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:22.725569 (+ 1192us) server_negotiation.cc:407] Connection header received
0620 18:42:22.727332 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:22.727339 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:22.727433 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:22.727567 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:22.729231 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:22.730176 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:22.732042 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:22.732772 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:22.732914 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:22.736441 (+ 3527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:22.736446 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:22.736449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:22.736478 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:22.739966 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:22.740690 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:22.740699 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:22.740705 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:22.740861 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:22.741257 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:22.741264 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:22.741270 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:22.741666 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:22.741822 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:22.742484 (+ 662us) server_negotiation.cc:299] Negotiation successful
0620 18:42:22.742741 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 18:42:22.743834 11960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:22.724697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:42269 (local address 127.5.225.130:60037)
0620 18:42:22.725407 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:22.725448 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:22.727059 (+ 1611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:22.727860 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:22.727867 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:22.728423 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:22.729079 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:22.729096 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:22.730408 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:22.730416 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:22.731891 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:22.731903 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:22.732976 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:22.732980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:22.733170 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:22.733842 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:22.733868 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:22.736220 (+ 2352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:22.740116 (+ 3896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:22.740123 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:22.740138 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:22.740565 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:22.740967 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:22.740970 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:22.740973 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:22.741150 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:22.741784 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:22.741790 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:22.742077 (+ 287us) client_negotiation.cc:769] Sending connection context
0620 18:42:22.742369 (+ 292us) client_negotiation.cc:240] Negotiation successful
0620 18:42:22.742742 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":279,"threads_started":1}
I20260620 18:42:22.745246 11958 heartbeater.cc:347] Connected to a master server at 127.5.225.190:42269
I20260620 18:42:22.745589 11958 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:22.746516 11958 heartbeater.cc:511] Master 127.5.225.190:42269 requested a full tablet report, sending...
I20260620 18:42:22.748106 11635 ts_manager.cc:194] Registered new tserver with Master: a02d68d710914ec3979c778bdb81ada6 (127.5.225.130:42515)
WARNING: no policy specified for kudu/127.5.225.131@KRBTEST.COM; defaulting to no policy
I20260620 18:42:22.748940 11635 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.130@KRBTEST.COM'} at 127.5.225.130:60037
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab.
Entry for principal kudu/127.5.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab.
I20260620 18:42:22.774868 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.131:0
--local_ip_for_outbound_sockets=127.5.225.131
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:42269
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:35911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:23.094445 11963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:23.094784 11963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:23.094957 11963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:23.102269 11963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:23.102439 11963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:23.102645 11963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.131
I20260620 18:42:23.113011 11963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:35911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:42269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.131
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:23.114523 11963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:23.115849 11963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:23.128894 11971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:23.128873 11969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:23.129417 11968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:23.129734 11963 server_base.cc:1034] running on GCE node
I20260620 18:42:23.130422 11963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:23.131345 11963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:23.132578 11963 hybrid_clock.cc:648] HybridClock initialized: now 1781980943132535 us; error 67 us; skew 500 ppm
I20260620 18:42:23.132929 11963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:23 dist-test-slave-k6lg krb5kdc[11590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980943, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:23.136176 11963 init.cc:377] Logged in from keytab as kudu/127.5.225.131@KRBTEST.COM (short username kudu)
I20260620 18:42:23.137574 11963 webserver.cc:460] Webserver started at http://127.5.225.131:38883/ using document root <none> and password file <none>
I20260620 18:42:23.138358 11963 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:23.138543 11963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:23.138842 11963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:23.141655 11963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-2/data/instance:
uuid: "301e9c3b813f4b9ab57f2aeafa53e2d8"
format_stamp: "Formatted at 2026-06-20 18:42:23 on dist-test-slave-k6lg"
server_key: "2a5743eea4440bd4b8586e48653ea370"
server_key_iv: "89431165094fad70aeb5a02f44b16c68"
server_key_version: "encryptionkey@0"
I20260620 18:42:23.142463 11963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance:
uuid: "301e9c3b813f4b9ab57f2aeafa53e2d8"
format_stamp: "Formatted at 2026-06-20 18:42:23 on dist-test-slave-k6lg"
server_key: "2a5743eea4440bd4b8586e48653ea370"
server_key_iv: "89431165094fad70aeb5a02f44b16c68"
server_key_version: "encryptionkey@0"
I20260620 18:42:23.147323 11963 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:42:23.150801 11978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:23.152278 11963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 18:42:23.152452 11963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-2/wal
uuid: "301e9c3b813f4b9ab57f2aeafa53e2d8"
format_stamp: "Formatted at 2026-06-20 18:42:23 on dist-test-slave-k6lg"
server_key: "2a5743eea4440bd4b8586e48653ea370"
server_key_iv: "89431165094fad70aeb5a02f44b16c68"
server_key_version: "encryptionkey@0"
I20260620 18:42:23.152665 11963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:23.167260 11963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:23.171676 11963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:23.172019 11963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:23.173161 11963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:23.174676 11963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:23.174834 11963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:23.174973 11963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:23.175060 11963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:23.217434 11963 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.131:42337
I20260620 18:42:23.217478 12090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.131:42337 every 8 connection(s)
I20260620 18:42:23.218724 11963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
I20260620 18:42:23.225085 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 11963
I20260620 18:42:23.225292 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance
I20260620 18:42:23.225927 6022 external_mini_cluster.cc:1428] Setting key 007d69c48e6e21fe927244624f14895a
Jun 20 18:42:23 dist-test-slave-k6lg krb5kdc[11590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980943, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:42:23.239842 11826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:23.221126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:46111 (local address 127.5.225.190:42269)
0620 18:42:23.221292 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:23.221296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:23.222575 (+ 1279us) server_negotiation.cc:407] Connection header received
0620 18:42:23.225057 (+ 2482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:23.225065 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:23.225223 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:23.225344 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:23.227431 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.228326 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.229999 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.230478 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.230624 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:23.234001 (+ 3377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:23.234007 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:23.234010 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:23.234041 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:23.236739 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:23.237493 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:23.237500 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:23.237503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:23.237631 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:23.238074 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:23.238081 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:23.238084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:23.238423 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:23.238603 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:23.239245 (+ 642us) server_negotiation.cc:299] Negotiation successful
0620 18:42:23.239461 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 18:42:23.241106 12093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:23.221556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:42269 (local address 127.5.225.131:46111)
0620 18:42:23.222387 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:23.222439 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:23.224730 (+ 2291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:23.225737 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:23.225749 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:23.226505 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:23.227215 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:23.227259 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.228480 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.228485 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:23.229845 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:23.229857 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.230699 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.230708 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:23.230942 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:23.231631 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:23.231652 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:23.233781 (+ 2129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:23.236902 (+ 3121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:23.236911 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:23.236928 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:23.237365 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:23.237767 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:23.237772 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:23.237776 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:23.237959 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:23.238574 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:23.238582 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:23.238911 (+ 329us) client_negotiation.cc:769] Sending connection context
0620 18:42:23.239170 (+ 259us) client_negotiation.cc:240] Negotiation successful
0620 18:42:23.239689 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":329,"threads_started":1}
I20260620 18:42:23.242807 12091 heartbeater.cc:347] Connected to a master server at 127.5.225.190:42269
I20260620 18:42:23.243180 12091 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:23.243981 12091 heartbeater.cc:511] Master 127.5.225.190:42269 requested a full tablet report, sending...
I20260620 18:42:23.245391 11635 ts_manager.cc:194] Registered new tserver with Master: 301e9c3b813f4b9ab57f2aeafa53e2d8 (127.5.225.131:42337)
I20260620 18:42:23.246250 11635 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.131@KRBTEST.COM'} at 127.5.225.131:46111
I20260620 18:42:23.254551 6022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:42:23.257393 11824 heartbeater.cc:503] Master 127.5.225.190:42269 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/krb5kdc/test-admin.keytab.
Jun 20 18:42:23 dist-test-slave-k6lg krb5kdc[11590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:23.282326 6022 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 18:42:23.287899 12098 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:23.285193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36920 (local address 127.5.225.129:46301)
0620 18:42:23.285866 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:23.285874 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:23.285901 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:42:23.285964 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:23.285970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:23.286302 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:23.286448 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:23.287451 (+ 1003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36920: BlockingRecv error: recv got EOF from 127.0.0.1:36920 (error 108)
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":348,"threads_started":1}
W20260620 18:42:23.287884 6022 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/krb5kdc/krb5cc' of type FILE
Jun 20 18:42:23 dist-test-slave-k6lg krb5kdc[11590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:23.289820 6022 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 18:42:23 dist-test-slave-k6lg krb5kdc[11590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:23.291404 6022 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 18:42:23 dist-test-slave-k6lg krb5kdc[11590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.129@KRBTEST.COM
I20260620 18:42:23.307076 12098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:23.293105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36924 (local address 127.5.225.129:46301)
0620 18:42:23.293311 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:23.293316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:23.293392 (+ 76us) server_negotiation.cc:407] Connection header received
0620 18:42:23.293536 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:23.293540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:23.293596 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:23.293703 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:23.294847 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.295783 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.297122 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.297733 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.297847 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:23.301592 (+ 3745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:23.301603 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:23.301617 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:23.301650 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:23.304539 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:23.305160 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:23.305168 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:23.305170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:23.305283 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:23.305587 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:23.305590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:23.305592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:23.305953 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:23.306237 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:23.306509 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 18:42:23.306826 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260620 18:42:23.312548 12098 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:23.310959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36936 (local address 127.5.225.129:46301)
0620 18:42:23.311202 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:23.311207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:23.311394 (+ 187us) server_negotiation.cc:407] Connection header received
0620 18:42:23.311502 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:23.311506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:23.311569 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:23.311645 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:23.312316 (+ 671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36936: BlockingRecv error: recv got EOF from 127.0.0.1:36936 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
W20260620 18:42:23.312753 6022 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/krb5kdc/krb5cc' of type FILE
Jun 20 18:42:23 dist-test-slave-k6lg krb5kdc[11590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:23.314325 6022 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 18:42:23 dist-test-slave-k6lg krb5kdc[11590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:23.315587 6022 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 18:42:23 dist-test-slave-k6lg krb5kdc[11590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.129@KRBTEST.COM
I20260620 18:42:23.328181 12098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:23.316823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36948 (local address 127.5.225.129:46301)
0620 18:42:23.317001 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:23.317006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:23.317140 (+ 134us) server_negotiation.cc:407] Connection header received
0620 18:42:23.317249 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:23.317252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:23.317302 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:23.317377 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:23.318271 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.318914 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.320033 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.320462 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.320549 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:23.323737 (+ 3188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:23.323741 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:23.323744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:23.323772 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:23.326058 (+ 2286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:23.326635 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:23.326639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:23.326641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:23.326750 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:23.327064 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:23.327067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:23.327069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:23.327319 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:23.327504 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:23.327736 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 18:42:23.327919 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 18:42:23.331934 12098 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:23.330470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36952 (local address 127.5.225.129:46301)
0620 18:42:23.330706 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:23.330711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:23.330845 (+ 134us) server_negotiation.cc:407] Connection header received
0620 18:42:23.330946 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:23.330949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:23.331010 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:23.331084 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:23.331739 (+ 655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36952: BlockingRecv error: recv got EOF from 127.0.0.1:36952 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260620 18:42:23.332332 6022 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980894474392-6022-0/krb5kdc/krb5cc)
Jun 20 18:42:23 dist-test-slave-k6lg krb5kdc[11590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:23.333680 6022 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 18:42:23 dist-test-slave-k6lg krb5kdc[11590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:23.335315 6022 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 18:42:23 dist-test-slave-k6lg krb5kdc[11590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.129@KRBTEST.COM
I20260620 18:42:23.348353 12098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:23.336641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36958 (local address 127.5.225.129:46301)
0620 18:42:23.336848 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:23.336854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:23.336966 (+ 112us) server_negotiation.cc:407] Connection header received
0620 18:42:23.337069 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:23.337074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:23.337125 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:23.337217 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:23.338091 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.338954 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.340128 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.340524 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.340609 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:23.343834 (+ 3225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:23.343838 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:23.343841 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:23.343868 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:23.346143 (+ 2275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:23.346750 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:23.346753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:23.346755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:23.346894 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:23.347199 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:23.347202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:23.347204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:23.347445 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:23.347657 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:23.347865 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 18:42:23.348052 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 18:42:23.350096 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 11696
I20260620 18:42:23.360870 12098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:23.350783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36970 (local address 127.5.225.129:46301)
0620 18:42:23.350907 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:23.350910 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:23.351271 (+ 361us) server_negotiation.cc:407] Connection header received
0620 18:42:23.351376 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:23.351380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:23.351425 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:23.351496 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:23.352293 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.352909 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.353984 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.354443 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.354531 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:23.356339 (+ 1808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:23.356343 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:23.356347 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:23.356406 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:23.358953 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:23.359475 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:23.359478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:23.359480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:23.359589 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:23.359867 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:23.359870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:23.359872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:23.360085 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:23.360268 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:23.360493 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 18:42:23.360629 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 18:42:23.361459 11819 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:23.449031 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 11696
I20260620 18:42:23.465175 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 11830
Jun 20 18:42:23 dist-test-slave-k6lg krb5kdc[11590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.130@KRBTEST.COM
I20260620 18:42:23.480803 12111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:23.466102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36058 (local address 127.5.225.130:42515)
0620 18:42:23.466794 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:23.466801 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:23.466827 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:42:23.466889 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:23.466895 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:23.467106 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:23.467244 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:23.468405 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.469449 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.470907 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.471474 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.471574 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:23.475411 (+ 3837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:23.475422 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:23.475437 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:23.475476 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:23.478055 (+ 2579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:23.478756 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:23.478762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:23.478765 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:23.478886 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:23.479265 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:23.479275 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:23.479277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:23.479592 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:23.479907 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:23.480242 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 18:42:23.480493 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":403,"threads_started":1}
I20260620 18:42:23.482092 11953 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:23.563072 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 11830
I20260620 18:42:23.578613 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 11963
Jun 20 18:42:23 dist-test-slave-k6lg krb5kdc[11590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.131@KRBTEST.COM
I20260620 18:42:23.594316 12113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:23.579620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41576 (local address 127.5.225.131:42337)
0620 18:42:23.580340 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:23.580346 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:23.580375 (+ 29us) server_negotiation.cc:407] Connection header received
0620 18:42:23.580432 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:23.580438 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:23.580675 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:23.580796 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:23.581785 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.582883 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.584121 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:23.584721 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:23.584822 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:23.588832 (+ 4010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:23.588842 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:23.588856 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:23.588888 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:23.591603 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:23.592262 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:23.592276 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:23.592279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:23.592393 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:23.592763 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:23.592770 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:23.592773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:23.593100 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:23.593409 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:23.593700 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 18:42:23.593951 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":411,"threads_started":1}
I20260620 18:42:23.595640 12086 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:23.677731 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 11963
I20260620 18:42:23.693497 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 11603
I20260620 18:42:23.694631 11665 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:23.815737 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 11603
2026-06-20T18:42:23Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3875 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:42:23 dist-test-slave-k6lg krb5kdc[12118](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:42:23 dist-test-slave-k6lg krb5kdc[12118](info): set up 2 sockets
Jun 20 18:42:23 dist-test-slave-k6lg krb5kdc[12118](info): commencing operation
krb5kdc: starting...
W20260620 18:42:24.910938 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.012s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:42:25 dist-test-slave-k6lg krb5kdc[12118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:42:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:42:25Z Disabled control of system clock
WARNING: no policy specified for oryx/127.5.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.190.keytab.
Entry for principal oryx/127.5.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.190.keytab.
I20260620 18:42:25.118392 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:34625
--webserver_interface=127.5.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.190.keytab
--principal=oryx/127.5.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:42967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:34625
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:25.438982 12131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:25.439266 12131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:25.439433 12131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:25.446565 12131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:42:25.446673 12131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:25.446776 12131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:25.446878 12131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:42:25.446960 12131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:42:25.456737 12131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:42967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:34625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.190.keytab
--principal=oryx/127.5.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:34625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:25.458062 12131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:25.459419 12131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:25.471174 12136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:25.471150 12137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:25.471225 12139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:25.471940 12131 server_base.cc:1034] running on GCE node
I20260620 18:42:25.472779 12131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:25.474349 12131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:25.475549 12131 hybrid_clock.cc:648] HybridClock initialized: now 1781980945475509 us; error 39 us; skew 500 ppm
I20260620 18:42:25.475920 12131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:25 dist-test-slave-k6lg krb5kdc[12118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980945, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:25.479012 12131 init.cc:377] Logged in from keytab as oryx/127.5.225.190@KRBTEST.COM (short username oryx)
I20260620 18:42:25.480432 12131 webserver.cc:460] Webserver started at http://127.5.225.190:43297/ using document root <none> and password file <none>
I20260620 18:42:25.481068 12131 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:25.481176 12131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:25.481423 12131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:25.484364 12131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "6cf69d27a27142a9b0c07415bb23d249"
format_stamp: "Formatted at 2026-06-20 18:42:25 on dist-test-slave-k6lg"
server_key: "a10074cf73d1678690e805ae42184162"
server_key_iv: "4283efa1cd60dd13844f75be72e9a60c"
server_key_version: "encryptionkey@0"
I20260620 18:42:25.485083 12131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "6cf69d27a27142a9b0c07415bb23d249"
format_stamp: "Formatted at 2026-06-20 18:42:25 on dist-test-slave-k6lg"
server_key: "a10074cf73d1678690e805ae42184162"
server_key_iv: "4283efa1cd60dd13844f75be72e9a60c"
server_key_version: "encryptionkey@0"
I20260620 18:42:25.490141 12131 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:42:25.493566 12146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:25.494966 12131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 18:42:25.495137 12131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "6cf69d27a27142a9b0c07415bb23d249"
format_stamp: "Formatted at 2026-06-20 18:42:25 on dist-test-slave-k6lg"
server_key: "a10074cf73d1678690e805ae42184162"
server_key_iv: "4283efa1cd60dd13844f75be72e9a60c"
server_key_version: "encryptionkey@0"
I20260620 18:42:25.495348 12131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:25.514394 12131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:25.518544 12131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:25.518843 12131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:25.537452 12131 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:34625
I20260620 18:42:25.537485 12197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:34625 every 8 connection(s)
I20260620 18:42:25.538903 12131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:42:25.542763 12198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:25.547166 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 12131
I20260620 18:42:25.547391 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:42:25.548032 6022 external_mini_cluster.cc:1428] Setting key 8b2a5ee559fb4dacbac22f8468326b48
I20260620 18:42:25.553550 12198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249: Bootstrap starting.
I20260620 18:42:25.557258 12198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:25.558537 12198 log.cc:824] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:25.562292 12198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249: No bootstrap required, opened a new log
Jun 20 18:42:25 dist-test-slave-k6lg krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM
I20260620 18:42:25.573217 12201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:25.550006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42728 (local address 127.5.225.190:34625)
0620 18:42:25.550858 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:25.550869 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:25.550914 (+ 45us) server_negotiation.cc:407] Connection header received
0620 18:42:25.552594 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:25.552617 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:25.553040 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:25.553376 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:25.555223 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:25.556520 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:25.558390 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:25.559476 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:25.559700 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:25.564329 (+ 4629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:25.564340 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:25.564372 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:25.564431 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:25.568514 (+ 4083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:25.569249 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:25.569263 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:25.569273 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:25.569505 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:25.569875 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:25.569881 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:25.569885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:25.570667 (+ 782us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:25.570996 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:25.571341 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 18:42:25.571821 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":438,"threads_started":1}
I20260620 18:42:25.573235 12198 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } }
I20260620 18:42:25.573773 12198 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:25.573940 12198 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cf69d27a27142a9b0c07415bb23d249, State: Initialized, Role: FOLLOWER
I20260620 18:42:25.574685 12198 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } }
I20260620 18:42:25.574920 12198 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:42:25.575049 12198 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:42:25.575232 12198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:25.577490 12198 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } }
I20260620 18:42:25.577981 12198 leader_election.cc:302] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cf69d27a27142a9b0c07415bb23d249; no voters:
I20260620 18:42:25.578626 12198 leader_election.cc:288] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:42:25.578780 12203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:42:25.580078 12203 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [term 1 LEADER]: Becoming Leader. State: Replica: 6cf69d27a27142a9b0c07415bb23d249, State: Running, Role: LEADER
I20260620 18:42:25.580574 12203 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } }
I20260620 18:42:25.580895 12198 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:42:25.583350 12204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cf69d27a27142a9b0c07415bb23d249" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } }
I20260620 18:42:25.583323 12205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cf69d27a27142a9b0c07415bb23d249. Latest consensus state: current_term: 1 leader_uuid: "6cf69d27a27142a9b0c07415bb23d249" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } }
I20260620 18:42:25.583756 12204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:25.583854 12205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:25.584717 12210 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:42:25.591084 12210 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:42:25.601432 12210 catalog_manager.cc:1347] Generated new cluster ID: fdd42e8c2ff34c219f495835c8ada190
I20260620 18:42:25.601540 12210 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:42:25.610648 12210 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:42:25.611989 12210 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:42:25.619704 12210 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249: Generated new TSK 0
I20260620 18:42:25.620448 12210 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.5.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.129.keytab.
Entry for principal oryx/127.5.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.129.keytab.
I20260620 18:42:25.678081 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.129:0
--local_ip_for_outbound_sockets=127.5.225.129
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:34625
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.129.keytab
--principal=oryx/127.5.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:42967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:26.026458 12224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:26.026938 12224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:26.027055 12224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:26.034446 12224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:26.034566 12224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:26.034682 12224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.129
I20260620 18:42:26.044446 12224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:42967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.129.keytab
--principal=oryx/127.5.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:34625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.129
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:26.045713 12224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:26.047060 12224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:26.059732 12229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:26.059739 12230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:26.059739 12232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:26.061008 12224 server_base.cc:1034] running on GCE node
I20260620 18:42:26.061724 12224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:26.062841 12224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:26.064143 12224 hybrid_clock.cc:648] HybridClock initialized: now 1781980946064096 us; error 62 us; skew 500 ppm
I20260620 18:42:26.064493 12224 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:26 dist-test-slave-k6lg krb5kdc[12118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980946, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:26.067873 12224 init.cc:377] Logged in from keytab as oryx/127.5.225.129@KRBTEST.COM (short username oryx)
I20260620 18:42:26.069408 12224 webserver.cc:460] Webserver started at http://127.5.225.129:39769/ using document root <none> and password file <none>
I20260620 18:42:26.070142 12224 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:26.070339 12224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:26.070649 12224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:26.073741 12224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-0/data/instance:
uuid: "911d8be242834719ac569e965601a2e2"
format_stamp: "Formatted at 2026-06-20 18:42:26 on dist-test-slave-k6lg"
server_key: "911e27054016cb2af5f3e1c40a37c3cf"
server_key_iv: "cb517d62c1f74345c672fefebc338be0"
server_key_version: "encryptionkey@0"
I20260620 18:42:26.074657 12224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance:
uuid: "911d8be242834719ac569e965601a2e2"
format_stamp: "Formatted at 2026-06-20 18:42:26 on dist-test-slave-k6lg"
server_key: "911e27054016cb2af5f3e1c40a37c3cf"
server_key_iv: "cb517d62c1f74345c672fefebc338be0"
server_key_version: "encryptionkey@0"
I20260620 18:42:26.079952 12224 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:42:26.083992 12239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:26.085490 12224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:42:26.085680 12224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-0/wal
uuid: "911d8be242834719ac569e965601a2e2"
format_stamp: "Formatted at 2026-06-20 18:42:26 on dist-test-slave-k6lg"
server_key: "911e27054016cb2af5f3e1c40a37c3cf"
server_key_iv: "cb517d62c1f74345c672fefebc338be0"
server_key_version: "encryptionkey@0"
I20260620 18:42:26.085886 12224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:26.106953 12224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:26.111402 12224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:26.111708 12224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:26.112762 12224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:26.114087 12224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:26.114214 12224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:26.114362 12224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:26.114434 12224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:26.147186 12224 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.129:42219
I20260620 18:42:26.147243 12351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.129:42219 every 8 connection(s)
I20260620 18:42:26.148492 12224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
I20260620 18:42:26.150903 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 12224
I20260620 18:42:26.151142 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance
I20260620 18:42:26.151965 6022 external_mini_cluster.cc:1428] Setting key bb340d2f6a3ce100dfd9cbee201de9e5
Jun 20 18:42:26 dist-test-slave-k6lg krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980946, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.225.129@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM
I20260620 18:42:26.169728 12354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:26.151219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:56109 (local address 127.5.225.190:34625)
0620 18:42:26.151749 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:26.151756 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:26.152860 (+ 1104us) server_negotiation.cc:407] Connection header received
0620 18:42:26.154736 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:26.154745 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:26.154833 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:26.154953 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:26.156422 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:26.157596 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:26.159770 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:26.160184 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:26.160328 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:26.163967 (+ 3639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:26.163973 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:26.163976 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:26.164009 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:26.166665 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:26.167416 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:26.167426 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:26.167428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:26.167542 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:26.167980 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:26.167985 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:26.167988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:26.168261 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:26.168414 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:26.169133 (+ 719us) server_negotiation.cc:299] Negotiation successful
0620 18:42:26.169348 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":251,"threads_started":1}
WARNING: no policy specified for oryx/127.5.225.130@KRBTEST.COM; defaulting to no policy
I20260620 18:42:26.170806 12355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:26.151859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:34625 (local address 127.5.225.129:56109)
0620 18:42:26.152695 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:26.152745 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:26.154474 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:26.155161 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:26.155169 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:26.155703 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:26.156246 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:26.156263 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:26.157754 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:26.157762 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:26.159605 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:26.159618 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:26.160411 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:26.160420 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:26.160666 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:26.161417 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:26.161446 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:26.163737 (+ 2291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:26.166810 (+ 3073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:26.166819 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:26.166844 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:26.167277 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:26.167652 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:26.167658 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:26.167661 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:26.167860 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:26.168390 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:26.168398 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:26.168746 (+ 348us) client_negotiation.cc:769] Sending connection context
0620 18:42:26.169011 (+ 265us) client_negotiation.cc:240] Negotiation successful
0620 18:42:26.169562 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":329,"threads_started":1}
I20260620 18:42:26.172652 12352 heartbeater.cc:347] Connected to a master server at 127.5.225.190:34625
I20260620 18:42:26.172987 12352 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.225.130@KRBTEST.COM" created.
I20260620 18:42:26.173923 12352 heartbeater.cc:511] Master 127.5.225.190:34625 requested a full tablet report, sending...
I20260620 18:42:26.176520 12163 ts_manager.cc:194] Registered new tserver with Master: 911d8be242834719ac569e965601a2e2 (127.5.225.129:42219)
I20260620 18:42:26.178933 12163 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.225.129@KRBTEST.COM'} at 127.5.225.129:56109
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.130.keytab.
Entry for principal oryx/127.5.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.130.keytab.
I20260620 18:42:26.200871 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.130:0
--local_ip_for_outbound_sockets=127.5.225.130
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:34625
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.130.keytab
--principal=oryx/127.5.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:42967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:26.526096 12358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:26.526453 12358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:26.526655 12358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:26.533866 12358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:26.533967 12358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:26.534142 12358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.130
I20260620 18:42:26.544142 12358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:42967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.130.keytab
--principal=oryx/127.5.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:34625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.130
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:26.545395 12358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:26.546763 12358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:26.559908 12363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:26.560025 12364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:26.559954 12366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:26.560925 12358 server_base.cc:1034] running on GCE node
I20260620 18:42:26.561563 12358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:26.562566 12358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:26.563779 12358 hybrid_clock.cc:648] HybridClock initialized: now 1781980946563725 us; error 41 us; skew 500 ppm
I20260620 18:42:26.564116 12358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:26 dist-test-slave-k6lg krb5kdc[12118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980946, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:26.567348 12358 init.cc:377] Logged in from keytab as oryx/127.5.225.130@KRBTEST.COM (short username oryx)
I20260620 18:42:26.568738 12358 webserver.cc:460] Webserver started at http://127.5.225.130:35899/ using document root <none> and password file <none>
I20260620 18:42:26.569422 12358 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:26.569540 12358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:26.569834 12358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:26.572860 12358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-1/data/instance:
uuid: "762e84cc86204351bc255b1e796e833f"
format_stamp: "Formatted at 2026-06-20 18:42:26 on dist-test-slave-k6lg"
server_key: "fff5fe99ed51ba2e3a388256c2ca97f4"
server_key_iv: "0d43efc4d3a2fe6df64c23c7460d05d4"
server_key_version: "encryptionkey@0"
I20260620 18:42:26.573618 12358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance:
uuid: "762e84cc86204351bc255b1e796e833f"
format_stamp: "Formatted at 2026-06-20 18:42:26 on dist-test-slave-k6lg"
server_key: "fff5fe99ed51ba2e3a388256c2ca97f4"
server_key_iv: "0d43efc4d3a2fe6df64c23c7460d05d4"
server_key_version: "encryptionkey@0"
I20260620 18:42:26.578881 12358 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 18:42:26.582474 12373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:26.583851 12358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 18:42:26.584007 12358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-1/wal
uuid: "762e84cc86204351bc255b1e796e833f"
format_stamp: "Formatted at 2026-06-20 18:42:26 on dist-test-slave-k6lg"
server_key: "fff5fe99ed51ba2e3a388256c2ca97f4"
server_key_iv: "0d43efc4d3a2fe6df64c23c7460d05d4"
server_key_version: "encryptionkey@0"
I20260620 18:42:26.584209 12358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:26.613915 12358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:26.618095 12358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:26.618407 12358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:26.619547 12358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:26.621186 12358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:26.621260 12358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:26.621388 12358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:26.621444 12358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:26.654973 12358 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.130:45529
I20260620 18:42:26.655012 12485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.130:45529 every 8 connection(s)
I20260620 18:42:26.656418 12358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
I20260620 18:42:26.663136 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 12358
I20260620 18:42:26.663322 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance
I20260620 18:42:26.663928 6022 external_mini_cluster.cc:1428] Setting key d5dfd4b3c77b90041012a87ce8e0bdde
Jun 20 18:42:26 dist-test-slave-k6lg krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980946, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.225.130@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM
I20260620 18:42:26.677613 12354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:26.659109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:56263 (local address 127.5.225.190:34625)
0620 18:42:26.659299 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:26.659303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:26.660469 (+ 1166us) server_negotiation.cc:407] Connection header received
0620 18:42:26.662221 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:26.662227 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:26.662322 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:26.662432 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:26.664428 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:26.665171 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:26.667000 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:26.667840 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:26.667982 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:26.671636 (+ 3654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:26.671646 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:26.671655 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:26.671702 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:26.674397 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:26.675325 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:26.675328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:26.675331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:26.675450 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:26.675930 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:26.675934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:26.675936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:26.676188 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:26.676304 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:26.677197 (+ 893us) server_negotiation.cc:299] Negotiation successful
0620 18:42:26.677375 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:42:26.679207 12488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:26.659623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:34625 (local address 127.5.225.130:56263)
0620 18:42:26.660315 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:26.660354 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:26.661939 (+ 1585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:26.662655 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:26.662662 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:26.663440 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:26.664227 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:26.664252 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:26.665319 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:26.665325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:26.666848 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:26.666860 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:26.668056 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:26.668060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:26.668263 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:26.668946 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:26.668967 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:26.671378 (+ 2411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:26.674531 (+ 3153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:26.674543 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:26.674564 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:26.675205 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:26.675561 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:26.675568 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:26.675575 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:26.675820 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:26.676321 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:26.676332 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:26.676797 (+ 465us) client_negotiation.cc:769] Sending connection context
0620 18:42:26.677128 (+ 331us) client_negotiation.cc:240] Negotiation successful
0620 18:42:26.677681 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":274,"threads_started":1}
I20260620 18:42:26.680810 12486 heartbeater.cc:347] Connected to a master server at 127.5.225.190:34625
I20260620 18:42:26.681157 12486 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:26.681995 12486 heartbeater.cc:511] Master 127.5.225.190:34625 requested a full tablet report, sending...
I20260620 18:42:26.683568 12163 ts_manager.cc:194] Registered new tserver with Master: 762e84cc86204351bc255b1e796e833f (127.5.225.130:45529)
I20260620 18:42:26.684742 12163 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.225.130@KRBTEST.COM'} at 127.5.225.130:56263
WARNING: no policy specified for oryx/127.5.225.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.225.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.131.keytab.
Entry for principal oryx/127.5.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.131.keytab.
I20260620 18:42:26.712868 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.131:0
--local_ip_for_outbound_sockets=127.5.225.131
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:34625
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.131.keytab
--principal=oryx/127.5.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:42967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:27.037818 12491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:27.038107 12491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:27.038271 12491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:27.045605 12491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:27.045718 12491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:27.045840 12491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.131
I20260620 18:42:27.055797 12491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:42967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.131.keytab
--principal=oryx/127.5.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:34625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.131
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:27.057158 12491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:27.058534 12491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:27.073470 12499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:27.073544 12497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:27.073513 12496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:27.074291 12491 server_base.cc:1034] running on GCE node
I20260620 18:42:27.074901 12491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:27.075877 12491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:27.077100 12491 hybrid_clock.cc:648] HybridClock initialized: now 1781980947077028 us; error 61 us; skew 500 ppm
I20260620 18:42:27.077431 12491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:27 dist-test-slave-k6lg krb5kdc[12118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980947, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:27.080871 12491 init.cc:377] Logged in from keytab as oryx/127.5.225.131@KRBTEST.COM (short username oryx)
I20260620 18:42:27.082440 12491 webserver.cc:460] Webserver started at http://127.5.225.131:40319/ using document root <none> and password file <none>
I20260620 18:42:27.083161 12491 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:27.083310 12491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:27.083621 12491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:27.086612 12491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-2/data/instance:
uuid: "3417abed2a0446919a8150bec84fbafc"
format_stamp: "Formatted at 2026-06-20 18:42:27 on dist-test-slave-k6lg"
server_key: "9629e60360e84005c6a684b42298859b"
server_key_iv: "c991f75e20258c4123f5f96692db1664"
server_key_version: "encryptionkey@0"
I20260620 18:42:27.087409 12491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance:
uuid: "3417abed2a0446919a8150bec84fbafc"
format_stamp: "Formatted at 2026-06-20 18:42:27 on dist-test-slave-k6lg"
server_key: "9629e60360e84005c6a684b42298859b"
server_key_iv: "c991f75e20258c4123f5f96692db1664"
server_key_version: "encryptionkey@0"
I20260620 18:42:27.092862 12491 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 18:42:27.096380 12506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:27.097944 12491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:42:27.098090 12491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-2/wal
uuid: "3417abed2a0446919a8150bec84fbafc"
format_stamp: "Formatted at 2026-06-20 18:42:27 on dist-test-slave-k6lg"
server_key: "9629e60360e84005c6a684b42298859b"
server_key_iv: "c991f75e20258c4123f5f96692db1664"
server_key_version: "encryptionkey@0"
I20260620 18:42:27.098392 12491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:27.127269 12491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:27.131853 12491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:27.132140 12491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:27.133096 12491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:27.134506 12491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:27.134577 12491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:27.134720 12491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:27.134776 12491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:27.167609 12491 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.131:42693
I20260620 18:42:27.167649 12618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.131:42693 every 8 connection(s)
I20260620 18:42:27.168881 12491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
I20260620 18:42:27.172719 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 12491
I20260620 18:42:27.172909 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance
I20260620 18:42:27.173453 6022 external_mini_cluster.cc:1428] Setting key bc03cc294ac26a2fec8cae9e08b2afb1
I20260620 18:42:27.182260 12352 heartbeater.cc:503] Master 127.5.225.190:34625 was elected leader, sending a full tablet report...
Jun 20 18:42:27 dist-test-slave-k6lg krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980947, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.225.131@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM
I20260620 18:42:27.190110 12354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:27.171384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:57473 (local address 127.5.225.190:34625)
0620 18:42:27.171602 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:27.171609 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:27.172972 (+ 1363us) server_negotiation.cc:407] Connection header received
0620 18:42:27.175335 (+ 2363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:27.175358 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:27.175481 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:27.175581 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:27.177302 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:27.178209 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:27.179814 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:27.180354 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:27.180471 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:27.184113 (+ 3642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:27.184118 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:27.184121 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:27.184150 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:27.186657 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:27.187399 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:27.187404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:27.187409 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:27.187550 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:27.188029 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:27.188035 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:27.188039 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:27.188333 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:27.188538 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:27.189541 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0620 18:42:27.189754 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 18:42:27.191190 12621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:27.171944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:34625 (local address 127.5.225.131:57473)
0620 18:42:27.172788 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:27.172843 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:27.175024 (+ 2181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:27.175862 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:27.175871 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:27.176480 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:27.177157 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:27.177173 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:27.178362 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:27.178366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:27.179658 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:27.179669 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:27.180549 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:27.180554 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:27.180760 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:27.181444 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:27.181466 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:27.183940 (+ 2474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:27.186790 (+ 2850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:27.186796 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:27.186826 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:27.187241 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:27.187672 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:27.187678 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:27.187681 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:27.187917 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:27.188570 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:27.188578 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:27.189103 (+ 525us) client_negotiation.cc:769] Sending connection context
0620 18:42:27.189436 (+ 333us) client_negotiation.cc:240] Negotiation successful
0620 18:42:27.189999 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":311,"threads_started":1}
I20260620 18:42:27.192509 12619 heartbeater.cc:347] Connected to a master server at 127.5.225.190:34625
I20260620 18:42:27.192833 12619 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:27.193542 12619 heartbeater.cc:511] Master 127.5.225.190:34625 requested a full tablet report, sending...
I20260620 18:42:27.194957 12163 ts_manager.cc:194] Registered new tserver with Master: 3417abed2a0446919a8150bec84fbafc (127.5.225.131:42693)
I20260620 18:42:27.195751 12163 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.225.131@KRBTEST.COM'} at 127.5.225.131:57473
I20260620 18:42:27.201107 6022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.5.225.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.225.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.225.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.189.keytab.
Entry for principal oryx/127.5.225.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.189.keytab.
I20260620 18:42:27.249861 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.189:37095
--webserver_interface=127.5.225.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.189.keytab
--principal=oryx/127.5.225.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:42967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:34625,127.5.225.189:37095
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:27.576440 12624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:27.576720 12624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:27.576887 12624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:27.584277 12624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:42:27.584383 12624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:27.584481 12624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:27.584576 12624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:42:27.584656 12624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:42:27.594915 12624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:42967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:34625,127.5.225.189:37095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.189.keytab
--principal=oryx/127.5.225.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.189:37095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.225.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
W20260620 18:42:27.595288 12624 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.5.225.190:34625,127.5.225.189:37095'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 18:42:27.596427 12624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:27.597746 12624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:27.609712 12629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:27.609805 12630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:27.609829 12632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:27.610812 12624 server_base.cc:1034] running on GCE node
I20260620 18:42:27.611642 12624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:27.612999 12624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:27.614271 12624 hybrid_clock.cc:648] HybridClock initialized: now 1781980947614215 us; error 36 us; skew 500 ppm
I20260620 18:42:27.614599 12624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:27 dist-test-slave-k6lg krb5kdc[12118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980947, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.225.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:27.617695 12624 init.cc:377] Logged in from keytab as oryx/127.5.225.189@KRBTEST.COM (short username oryx)
I20260620 18:42:27.619323 12624 webserver.cc:460] Webserver started at http://127.5.225.189:46761/ using document root <none> and password file <none>
I20260620 18:42:27.620049 12624 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:27.620178 12624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:27.620494 12624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:27.623425 12624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-1/data/instance:
uuid: "45a9ce1454f24d488dd7295235c75786"
format_stamp: "Formatted at 2026-06-20 18:42:27 on dist-test-slave-k6lg"
server_key: "864df10bda115fe75b704a3e39799bb3"
server_key_iv: "74aea5d2d7cca8a9d4ea0b5f94f926d2"
server_key_version: "encryptionkey@0"
I20260620 18:42:27.624181 12624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-1/wal/instance:
uuid: "45a9ce1454f24d488dd7295235c75786"
format_stamp: "Formatted at 2026-06-20 18:42:27 on dist-test-slave-k6lg"
server_key: "864df10bda115fe75b704a3e39799bb3"
server_key_iv: "74aea5d2d7cca8a9d4ea0b5f94f926d2"
server_key_version: "encryptionkey@0"
I20260620 18:42:27.629366 12624 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260620 18:42:27.632807 12639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:27.634423 12624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:42:27.634644 12624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-1/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-1/wal
uuid: "45a9ce1454f24d488dd7295235c75786"
format_stamp: "Formatted at 2026-06-20 18:42:27 on dist-test-slave-k6lg"
server_key: "864df10bda115fe75b704a3e39799bb3"
server_key_iv: "74aea5d2d7cca8a9d4ea0b5f94f926d2"
server_key_version: "encryptionkey@0"
I20260620 18:42:27.634864 12624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:27.655516 12624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:27.660041 12624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:27.660305 12624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:27.679121 12624 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.189:37095
I20260620 18:42:27.679131 12690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.189:37095 every 8 connection(s)
I20260620 18:42:27.680449 12624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-1/data/info.pb
I20260620 18:42:27.684362 12691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:27.687387 12486 heartbeater.cc:503] Master 127.5.225.190:34625 was elected leader, sending a full tablet report...
I20260620 18:42:27.690728 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 12624
I20260620 18:42:27.691642 12691 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:42:27 dist-test-slave-k6lg krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980947, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.225.189@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM
Jun 20 18:42:27 dist-test-slave-k6lg krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.225.189@KRBTEST.COM
I20260620 18:42:27.712225 12695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:27.694378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59934 (local address 127.5.225.190:34625)
0620 18:42:27.694811 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:27.694818 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:27.695658 (+ 840us) server_negotiation.cc:407] Connection header received
0620 18:42:27.696525 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:27.696530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:27.696611 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:27.696736 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:27.698440 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:27.699612 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:27.701668 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:27.702202 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:27.702324 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:27.705572 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:27.705577 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:27.705581 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:27.705613 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:27.708813 (+ 3200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:27.709860 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:27.709864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:27.709866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:27.709989 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:27.710459 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:27.710462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:27.710464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:27.710815 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:27.710967 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:27.711718 (+ 751us) server_negotiation.cc:299] Negotiation successful
0620 18:42:27.711956 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":506,"threads_started":1}
I20260620 18:42:27.713518 12696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:27.694621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:34625 (local address 127.0.0.1:59934)
0620 18:42:27.695503 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:27.695541 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:27.696298 (+ 757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:27.696818 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:27.696825 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:27.697670 (+ 845us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:27.698253 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:27.698278 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:27.699799 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:27.699807 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:27.701522 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:27.701537 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:27.702396 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:27.702402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:27.702611 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:27.703347 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:27.703367 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:27.705383 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:27.708953 (+ 3570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:27.708963 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:27.708974 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:27.709744 (+ 770us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:27.710101 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:27.710107 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:27.710110 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:27.710351 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:27.710943 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:27.710950 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:27.711236 (+ 286us) client_negotiation.cc:769] Sending connection context
0620 18:42:27.711594 (+ 358us) client_negotiation.cc:240] Negotiation successful
0620 18:42:27.712004 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"mutex_wait_us":41,"thread_start_us":446,"threads_started":1}
I20260620 18:42:27.713624 12694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:27.693239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47336 (local address 127.5.225.189:37095)
0620 18:42:27.694480 (+ 1241us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:27.694490 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:27.694525 (+ 35us) server_negotiation.cc:407] Connection header received
0620 18:42:27.696282 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:27.696302 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:27.696593 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:27.696783 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:27.698687 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:27.700203 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:27.701972 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:27.702825 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:27.702971 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:27.707468 (+ 4497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:27.707477 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:27.707489 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:27.707515 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:27.710676 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:27.711363 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:27.711371 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:27.711375 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:27.711507 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:27.711958 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:27.711963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:27.711966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:27.712348 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:27.712636 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:27.712987 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 18:42:27.713215 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1052,"thread_start_us":824,"threads_started":1}
I20260620 18:42:27.715390 12691 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.225.189" port: 37095 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:42:27 dist-test-slave-k6lg krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980947, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.225.189@KRBTEST.COM for oryx/127.5.225.189@KRBTEST.COM
I20260620 18:42:27.727123 12696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:27.716286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:37095 (local address 127.0.0.1:47338)
0620 18:42:27.716586 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:27.716602 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:27.716674 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:27.717005 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:27.717008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:27.717292 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:27.717414 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:27.717421 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:27.718411 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:27.718414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:27.719288 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:27.719298 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:27.719899 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:27.719903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:27.720052 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:27.720508 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:27.720556 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:27.722428 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:27.725143 (+ 2715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:27.725146 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:27.725149 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:27.725496 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:27.725829 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:27.725832 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:27.725834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:27.725947 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:27.726457 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:27.726460 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:27.726574 (+ 114us) client_negotiation.cc:769] Sending connection context
0620 18:42:27.726684 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 18:42:27.726864 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174}
I20260620 18:42:27.727195 12694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:27.716260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47338 (local address 127.5.225.189:37095)
0620 18:42:27.716812 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:27.716816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:27.716825 (+ 9us) server_negotiation.cc:407] Connection header received
0620 18:42:27.716851 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:27.716854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:27.716894 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:27.716953 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:27.717533 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:27.718294 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:27.719405 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:27.719772 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:27.719865 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:27.722581 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:27.722585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:27.722588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:27.722614 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:27.725026 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:27.725596 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:27.725600 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:27.725602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:27.725736 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:27.726032 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:27.726034 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:27.726036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:27.726345 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:27.726451 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:27.726786 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 18:42:27.726968 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":467}
I20260620 18:42:27.731662 12691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786: Bootstrap starting.
I20260620 18:42:27.734568 12691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:27.735476 12691 log.cc:824] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:27.737833 12691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786: No bootstrap required, opened a new log
I20260620 18:42:27.744830 12691 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 37095 } }
I20260620 18:42:27.745115 12691 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:27.745194 12691 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45a9ce1454f24d488dd7295235c75786, State: Initialized, Role: FOLLOWER
I20260620 18:42:27.745791 12691 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 37095 } }
I20260620 18:42:27.746651 12698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 37095 } } }
I20260620 18:42:27.746974 12698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:42:27.747593 12691 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 18:42:27.756188 12709 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:42:27.756362 12709 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:42:27.758930 12624 master_runner.cc:428] Detected that this master 45a9ce1454f24d488dd7295235c75786 is joining an existing cluster
I20260620 18:42:27.759019 12624 master_runner.cc:432] Initiating AddMaster RPC to add 127.5.225.189:37095
I20260620 18:42:27.780452 12695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:27.762863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59944 (local address 127.5.225.190:34625)
0620 18:42:27.763060 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:27.763066 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:27.764218 (+ 1152us) server_negotiation.cc:407] Connection header received
0620 18:42:27.764431 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:27.764435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:27.764563 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:27.764665 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:27.766254 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:27.767409 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:27.769528 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:27.770368 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:27.770488 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:27.773372 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:27.773382 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:27.773389 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:27.773438 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:27.777218 (+ 3780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:27.778003 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:27.778012 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:27.778018 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:27.778214 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:27.778694 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:27.778701 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:27.778704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:27.779066 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:27.779197 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:27.779794 (+ 597us) server_negotiation.cc:299] Negotiation successful
0620 18:42:27.780046 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 18:42:27.781543 12716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:27.763140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:34625 (local address 127.0.0.1:59944)
0620 18:42:27.764116 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:27.764131 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:27.764247 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:27.764879 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:27.764886 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:27.765524 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:27.766045 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:27.766054 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:27.767573 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:27.767578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:27.769345 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:27.769362 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:27.770815 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:27.770823 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:27.771077 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:27.771815 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:27.771843 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:27.773155 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:27.777331 (+ 4176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:27.777338 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:27.777342 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:27.777855 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:27.778370 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:27.778374 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:27.778377 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:27.778554 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:27.779286 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:27.779291 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:27.779502 (+ 211us) client_negotiation.cc:769] Sending connection context
0620 18:42:27.780721 (+ 1219us) client_negotiation.cc:240] Negotiation successful
0620 18:42:27.781055 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":861,"mutex_wait_us":65,"thread_start_us":684,"threads_started":1}
I20260620 18:42:27.782280 12694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:27.763101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47354 (local address 127.5.225.189:37095)
0620 18:42:27.763265 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:27.763271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:27.764106 (+ 835us) server_negotiation.cc:407] Connection header received
0620 18:42:27.764318 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:27.764323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:27.764408 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:27.764522 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:27.767592 (+ 3070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:27.768736 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:27.770752 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:27.771895 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:27.772002 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:27.774791 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:27.774797 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:27.774801 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:27.774833 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:27.779163 (+ 4330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:27.779900 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:27.779904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:27.779910 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:27.780046 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:27.780420 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:27.780425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:27.780429 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:27.780864 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:27.780991 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:27.781552 (+ 561us) server_negotiation.cc:299] Negotiation successful
0620 18:42:27.781869 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":39}
I20260620 18:42:27.783008 12715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:27.762968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:37095 (local address 127.0.0.1:47354)
0620 18:42:27.763977 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:27.764002 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:27.764165 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:27.764557 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:27.764561 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:27.765414 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:27.766028 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:27.766045 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:27.768953 (+ 2908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:27.768960 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:27.770606 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:27.770620 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:27.772313 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:27.772321 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:27.772527 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:27.773219 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:27.773241 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:27.774613 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:27.779288 (+ 4675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:27.779292 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:27.779295 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:27.779788 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:27.780171 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:27.780175 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:27.780178 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:27.780317 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:27.781122 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:27.781127 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:27.781297 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 18:42:27.782224 (+ 927us) client_negotiation.cc:240] Negotiation successful
0620 18:42:27.782540 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"mutex_wait_us":21,"spinlock_wait_cycles":29056,"thread_start_us":410,"threads_started":1}
WARNING: no policy specified for oryx/127.5.225.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.225.188@KRBTEST.COM" created.
Jun 20 18:42:27 dist-test-slave-k6lg krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980945, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.225.190@KRBTEST.COM for oryx/127.5.225.189@KRBTEST.COM
I20260620 18:42:27.809764 12694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:27.788468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47360 (local address 127.5.225.189:37095)
0620 18:42:27.788687 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:27.788694 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:27.794877 (+ 6183us) server_negotiation.cc:407] Connection header received
0620 18:42:27.795239 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:27.795244 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:27.795347 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:27.795456 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:27.796617 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:27.797438 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:27.799106 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:27.799624 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:27.799777 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:27.803159 (+ 3382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:27.803167 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:27.803172 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:27.803207 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:27.807009 (+ 3802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:27.807688 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:27.807694 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:27.807697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:27.807833 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:27.808248 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:27.808252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:27.808256 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:27.808513 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:27.808685 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:27.809146 (+ 461us) server_negotiation.cc:299] Negotiation successful
0620 18:42:27.809374 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260620 18:42:27.809814 12718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:27.788509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:37095 (local address 127.0.0.1:47360)
0620 18:42:27.794741 (+ 6232us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:27.794787 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:27.795008 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:27.795513 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:27.795520 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:27.796252 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:27.796459 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:27.796468 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:27.797570 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:27.797575 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:27.798955 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:27.798971 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:27.799802 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:27.799808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:27.800007 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:27.800698 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:27.800721 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:27.802977 (+ 2256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:27.807144 (+ 4167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:27.807156 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:27.807163 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:27.807552 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:27.807945 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:27.807950 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:27.807954 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:27.808121 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:27.808647 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:27.808656 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:27.808856 (+ 200us) client_negotiation.cc:769] Sending connection context
0620 18:42:27.809042 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 18:42:27.809374 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5995,"thread_start_us":5833,"threads_started":1}
I20260620 18:42:27.811885 12163 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.5.225.189:37095: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "45a9ce1454f24d488dd7295235c75786"
member_type: NON_VOTER
last_known_addr {
host: "127.5.225.189"
port: 37095
}
attrs {
promote: true
}
}
dest_uuid: "6cf69d27a27142a9b0c07415bb23d249"
cas_config_opid_index: -1
I20260620 18:42:27.812829 12163 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: NON_VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: true } }
I20260620 18:42:27.821308 12722 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 45a9ce1454f24d488dd7295235c75786 (127.5.225.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: NON_VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: true } } }
I20260620 18:42:27.822754 12666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:27.826637 12666 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [term 1 FOLLOWER]: Refusing update from remote peer 6cf69d27a27142a9b0c07415bb23d249: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 18:42:27.826692 12722 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.5.225.189:37095
I20260620 18:42:27.827428 12721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6cf69d27a27142a9b0c07415bb23d249" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: NON_VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: true } } }
I20260620 18:42:27.827711 12624 master.cc:537] Master@127.5.225.189:37095 shutting down...
I20260620 18:42:27.827804 12721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.225.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.188.keytab.
Entry for principal oryx/127.5.225.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.188.keytab.
I20260620 18:42:27.830806 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.188:45955
--webserver_interface=127.5.225.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.188.keytab
--principal=oryx/127.5.225.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:42967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:34625,127.5.225.189:37095,127.5.225.188:45955
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:42:27.841830 12722 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: NON_VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:42:27.843094 12624 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:42:27.843713 12624 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:42:27.844010 12624 tablet_replica.cc:330] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786: stopping tablet replica
W20260620 18:42:27.845145 12148 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 18:42:27.848721 12148 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 -> Peer 45a9ce1454f24d488dd7295235c75786 (127.5.225.189:37095): Couldn't send request to peer 45a9ce1454f24d488dd7295235c75786. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 18:42:27.852833 12624 master.cc:552] Master@127.5.225.189:37095 shutdown complete.
I20260620 18:42:27.852942 12624 master_runner.cc:305] Clearing existing system tablet
I20260620 18:42:27.853314 12624 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:27.853428 12624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:27.855676 12624 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 18:42:27.857197 12727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:27.857493 12624 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:27.857774 12624 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:42:27.857863 12624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-1/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-1/wal
uuid: "45a9ce1454f24d488dd7295235c75786"
format_stamp: "Formatted at 2026-06-20 18:42:27 on dist-test-slave-k6lg"
server_key: "864df10bda115fe75b704a3e39799bb3"
server_key_iv: "74aea5d2d7cca8a9d4ea0b5f94f926d2"
server_key_version: "encryptionkey@0"
I20260620 18:42:27.861706 12624 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:42:27.866758 12624 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 18:42:27.866940 12624 log.cc:1196] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 18:42:27.867339 12624 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786: Deleting consensus metadata
I20260620 18:42:27.867595 12624 master_runner.cc:315] Copying system tablet from 127.5.225.190:34625
I20260620 18:42:27.870469 12624 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786: tablet copy: Beginning tablet copy session from remote peer at address 127.5.225.190:34625
I20260620 18:42:27.885887 12695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:27.871226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59954 (local address 127.5.225.190:34625)
0620 18:42:27.871440 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:27.871447 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:27.874692 (+ 3245us) server_negotiation.cc:407] Connection header received
0620 18:42:27.874902 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:27.874907 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:27.874986 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:27.875092 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:27.875865 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:27.876577 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:27.877822 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:27.878324 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:27.878453 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:27.880326 (+ 1873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:27.880332 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:27.880337 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:27.880376 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:27.883361 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:27.883946 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:27.883952 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:27.883956 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:27.884086 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:27.884457 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:27.884464 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:27.884467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:27.884744 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:27.884901 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:27.885303 (+ 402us) server_negotiation.cc:299] Negotiation successful
0620 18:42:27.885502 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:42:27.885921 12733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:27.871196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:34625 (local address 127.0.0.1:59954)
0620 18:42:27.874564 (+ 3368us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:27.874588 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:27.874749 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:27.875116 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:27.875121 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:27.875520 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:27.875711 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:27.875720 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:27.876739 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:27.876744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:27.877666 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:27.877678 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:27.878476 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:27.878482 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:27.878687 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:27.879444 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:27.879462 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:27.880105 (+ 643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:27.883472 (+ 3367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:27.883478 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:27.883482 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:27.883828 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:27.884212 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:27.884217 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:27.884221 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:27.884352 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:27.884871 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:27.884875 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:27.885044 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 18:42:27.885215 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 18:42:27.885503 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3243,"thread_start_us":3157,"threads_started":1}
I20260620 18:42:27.886999 12183 tablet_copy_service.cc:140] P 6cf69d27a27142a9b0c07415bb23d249: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 45a9ce1454f24d488dd7295235c75786 ({username='oryx', principal='oryx/127.5.225.189@KRBTEST.COM'} at 127.0.0.1:59954)
I20260620 18:42:27.887377 12183 tablet_copy_service.cc:161] P 6cf69d27a27142a9b0c07415bb23d249: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 45a9ce1454f24d488dd7295235c75786 at {username='oryx', principal='oryx/127.5.225.189@KRBTEST.COM'} at 127.0.0.1:59954: session id = 45a9ce1454f24d488dd7295235c75786-00000000000000000000000000000000
I20260620 18:42:27.891185 12183 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 18:42:27.892971 12624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 18:42:27.897390 12624 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786: tablet copy: Starting download of 0 data blocks...
I20260620 18:42:27.897699 12624 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786: tablet copy: Starting download of 1 WAL segments...
I20260620 18:42:27.900251 12624 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 18:42:27.903548 12183 tablet_copy_service.cc:342] P 6cf69d27a27142a9b0c07415bb23d249: Request end of tablet copy session 45a9ce1454f24d488dd7295235c75786-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.5.225.189@KRBTEST.COM'} at 127.0.0.1:59954
I20260620 18:42:27.903723 12183 tablet_copy_service.cc:433] P 6cf69d27a27142a9b0c07415bb23d249: ending tablet copy session 45a9ce1454f24d488dd7295235c75786-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 45a9ce1454f24d488dd7295235c75786
I20260620 18:42:27.913434 12624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:27.916098 12734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:27.921331 12735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:27.925876 12624 server_base.cc:1034] running on GCE node
W20260620 18:42:27.926328 12737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:27.926733 12624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:27.927388 12624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:27.928596 12624 hybrid_clock.cc:648] HybridClock initialized: now 1781980947928516 us; error 65 us; skew 500 ppm
I20260620 18:42:27.928822 12624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:27 dist-test-slave-k6lg krb5kdc[12118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980947, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.225.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:27.930776 12624 init.cc:377] Logged in from keytab as oryx/127.5.225.189@KRBTEST.COM (short username oryx)
I20260620 18:42:27.932011 12624 webserver.cc:460] Webserver started at http://127.5.225.189:44611/ using document root <none> and password file <none>
I20260620 18:42:27.932323 12624 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:27.932420 12624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:27.934737 12624 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260620 18:42:27.936324 12744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:27.936746 12624 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20260620 18:42:27.936904 12624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-1/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-1/wal
uuid: "45a9ce1454f24d488dd7295235c75786"
format_stamp: "Formatted at 2026-06-20 18:42:27 on dist-test-slave-k6lg"
server_key: "864df10bda115fe75b704a3e39799bb3"
server_key_iv: "74aea5d2d7cca8a9d4ea0b5f94f926d2"
server_key_version: "encryptionkey@0"
I20260620 18:42:27.937055 12624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:27.952517 12624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:27.955926 12624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:27.979810 12624 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.189:37095
I20260620 18:42:27.979831 12795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.189:37095 every 8 connection(s)
I20260620 18:42:27.980870 12624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-1/data/info.pb
I20260620 18:42:27.988111 12796 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:42:27.993127 12796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786: Bootstrap starting.
I20260620 18:42:28.006140 12796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:42:28.006637 12796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786: Bootstrap complete.
I20260620 18:42:28.007454 12796 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: NON_VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: true } }
I20260620 18:42:28.007690 12796 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 45a9ce1454f24d488dd7295235c75786, State: Initialized, Role: LEARNER
I20260620 18:42:28.007926 12796 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: NON_VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: true } }
I20260620 18:42:28.008689 12799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: NON_VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: true } } }
I20260620 18:42:28.009008 12796 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:42:28.009053 12799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [sys.catalog]: This master's current role is: LEARNER
I20260620 18:42:28.019482 12810 catalog_manager.cc:1259] Loaded cluster ID: fdd42e8c2ff34c219f495835c8ada190
I20260620 18:42:28.019595 12810 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786: loading cluster ID for follower catalog manager: success
I20260620 18:42:28.022087 12810 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786: acquiring CA information for follower catalog manager: success
I20260620 18:42:28.023818 12810 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:42:28.198243 12619 heartbeater.cc:503] Master 127.5.225.190:34625 was elected leader, sending a full tablet report...
I20260620 18:42:28.237735 12718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:28.229136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:37095 (local address 127.0.0.1:47372)
0620 18:42:28.229417 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:28.229438 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:28.229596 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:28.230629 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:28.230635 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:28.230671 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:28.230881 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:28.230894 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:28.232992 (+ 2098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:28.232999 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:28.235090 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:28.235108 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:28.236838 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:28.236845 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:28.237070 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:28.237128 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 18:42:28.237226 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 18:42:28.237417 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260620 18:42:28.237780 12811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:28.229364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47372 (local address 127.5.225.189:37095)
0620 18:42:28.230199 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:28.230205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:28.230226 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:42:28.230305 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:28.230310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:28.230404 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:28.230557 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:28.231057 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:28.232824 (+ 1767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:28.235267 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:28.236643 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:28.236774 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:28.237015 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:28.237270 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 18:42:28.237415 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":460,"threads_started":1}
I20260620 18:42:28.239058 12771 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260620 18:42:28.239627 12799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cf69d27a27142a9b0c07415bb23d249. Latest consensus state: current_term: 1 leader_uuid: "6cf69d27a27142a9b0c07415bb23d249" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: NON_VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: true } } }
I20260620 18:42:28.239880 12799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [sys.catalog]: This master's current role is: LEARNER
W20260620 18:42:28.259723 12724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:28.260108 12724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:28.260267 12724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:28.268415 12724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:42:28.268549 12724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:28.268637 12724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:28.268721 12724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:42:28.268810 12724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:42:28.278807 12724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:42967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:34625,127.5.225.189:37095,127.5.225.188:45955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.188.keytab
--principal=oryx/127.5.225.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.188:45955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.225.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:28.280426 12724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:28.281999 12724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:28.293736 12819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:28.293722 12817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:28.293910 12816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:28.294504 12724 server_base.cc:1034] running on GCE node
I20260620 18:42:28.295219 12724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:28.296494 12724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:28.297739 12724 hybrid_clock.cc:648] HybridClock initialized: now 1781980948297691 us; error 73 us; skew 500 ppm
I20260620 18:42:28.298074 12724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:28 dist-test-slave-k6lg krb5kdc[12118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980948, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.225.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:28.301192 12724 init.cc:377] Logged in from keytab as oryx/127.5.225.188@KRBTEST.COM (short username oryx)
I20260620 18:42:28.302779 12724 webserver.cc:460] Webserver started at http://127.5.225.188:40119/ using document root <none> and password file <none>
I20260620 18:42:28.303437 12724 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:28.303571 12724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:28.303855 12724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:28.306720 12724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-2/data/instance:
uuid: "c2efcc70af1e4061b9d972d5b56de43d"
format_stamp: "Formatted at 2026-06-20 18:42:28 on dist-test-slave-k6lg"
server_key: "0fce1fee7760eb5a11a4206a02f99c08"
server_key_iv: "e261b20ccf25834217bcd971f89a3377"
server_key_version: "encryptionkey@0"
I20260620 18:42:28.307464 12724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-2/wal/instance:
uuid: "c2efcc70af1e4061b9d972d5b56de43d"
format_stamp: "Formatted at 2026-06-20 18:42:28 on dist-test-slave-k6lg"
server_key: "0fce1fee7760eb5a11a4206a02f99c08"
server_key_iv: "e261b20ccf25834217bcd971f89a3377"
server_key_version: "encryptionkey@0"
I20260620 18:42:28.312525 12724 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:42:28.316010 12826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:28.317299 12724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:42:28.317474 12724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-2/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-2/wal
uuid: "c2efcc70af1e4061b9d972d5b56de43d"
format_stamp: "Formatted at 2026-06-20 18:42:28 on dist-test-slave-k6lg"
server_key: "0fce1fee7760eb5a11a4206a02f99c08"
server_key_iv: "e261b20ccf25834217bcd971f89a3377"
server_key_version: "encryptionkey@0"
I20260620 18:42:28.317660 12724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:28.346855 12724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:28.351106 12724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:28.351382 12724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:28.369066 12724 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.188:45955
I20260620 18:42:28.369096 12877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.188:45955 every 8 connection(s)
I20260620 18:42:28.370440 12724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-2/data/info.pb
I20260620 18:42:28.374008 12878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:28.377457 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 12724
I20260620 18:42:28.381791 12878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:42:28 dist-test-slave-k6lg krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980948, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.225.188@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM
Jun 20 18:42:28 dist-test-slave-k6lg krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.225.188@KRBTEST.COM
I20260620 18:42:28.400583 12695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:28.384713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59970 (local address 127.5.225.190:34625)
0620 18:42:28.384913 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:28.384920 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:28.385263 (+ 343us) server_negotiation.cc:407] Connection header received
0620 18:42:28.385462 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:28.385468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:28.385554 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:28.385657 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:28.387149 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:28.387901 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:28.389595 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:28.390053 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:28.390233 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:28.394086 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:28.394095 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:28.394102 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:28.394144 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:28.396820 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:28.397769 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:28.397772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:28.397774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:28.397889 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:28.398337 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:28.398342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:28.398346 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:28.398701 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:28.398846 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:28.399867 (+ 1021us) server_negotiation.cc:299] Negotiation successful
0620 18:42:28.400161 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 18:42:28.401470 12881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:28.379647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60394 (local address 127.5.225.188:45955)
0620 18:42:28.380445 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:28.380456 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:28.380496 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:42:28.382324 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:28.382346 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:28.382730 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:28.383099 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:28.384564 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:28.386670 (+ 2106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:28.388100 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:28.389044 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:28.389245 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:28.394727 (+ 5482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:28.394734 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:28.394746 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:28.394772 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:28.398026 (+ 3254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:28.398821 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:28.398829 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:28.398832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:28.398996 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:28.399317 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:28.399321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:28.399326 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:28.399726 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:28.399999 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:28.400421 (+ 422us) server_negotiation.cc:299] Negotiation successful
0620 18:42:28.400654 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":544,"thread_start_us":318,"threads_started":1}
I20260620 18:42:28.401508 12882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:28.384499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:34625 (local address 127.0.0.1:59970)
0620 18:42:28.385130 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:28.385168 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:28.385335 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:28.385766 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:28.385773 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:28.386686 (+ 913us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:28.386984 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:28.386992 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:28.388044 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:28.388051 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:28.389436 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:28.389451 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:28.390346 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:28.390353 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:28.390550 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:28.391204 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:28.391238 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:28.393896 (+ 2658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:28.396953 (+ 3057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:28.396963 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:28.396976 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:28.397648 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:28.397984 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:28.397989 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:28.397992 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:28.398231 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:28.398825 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:28.398833 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:28.399252 (+ 419us) client_negotiation.cc:769] Sending connection context
0620 18:42:28.399588 (+ 336us) client_negotiation.cc:240] Negotiation successful
0620 18:42:28.400087 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":4,"thread_start_us":259,"threads_started":1}
I20260620 18:42:28.404036 12878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.225.189" port: 37095 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:42:28 dist-test-slave-k6lg krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980948, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.225.188@KRBTEST.COM for oryx/127.5.225.189@KRBTEST.COM
I20260620 18:42:28.415853 12882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:28.404793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:37095 (local address 127.0.0.1:47374)
0620 18:42:28.404971 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:28.404989 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:28.405073 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:28.405312 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:28.405315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:28.405754 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:28.405909 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:28.405917 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:28.406909 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:28.406912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:28.407808 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:28.407818 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:28.408438 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:28.408442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:28.408575 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:28.409217 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:28.409233 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:28.411188 (+ 1955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:28.413760 (+ 2572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:28.413763 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:28.413766 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:28.414219 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:28.414551 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:28.414555 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:28.414557 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:28.414722 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:28.415144 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:28.415161 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:28.415299 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 18:42:28.415406 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 18:42:28.415612 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 18:42:28.415867 12811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:28.404874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47374 (local address 127.5.225.189:37095)
0620 18:42:28.405066 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:28.405071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:28.405088 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:42:28.405148 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:28.405152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:28.405208 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:28.405286 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:28.406045 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:28.406807 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:28.407923 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:28.408320 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:28.408410 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:28.411365 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:28.411370 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:28.411373 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:28.411403 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:28.413643 (+ 2240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:28.414312 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:28.414316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:28.414318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:28.414432 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:28.414813 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:28.414816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:28.414818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:28.415058 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:28.415222 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:28.415481 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 18:42:28.415641 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:42:28.417128 12878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.225.188" port: 45955 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:42:28 dist-test-slave-k6lg krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980948, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.225.188@KRBTEST.COM for oryx/127.5.225.188@KRBTEST.COM
I20260620 18:42:28.428552 12881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:28.417795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60398 (local address 127.5.225.188:45955)
0620 18:42:28.417943 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:28.417949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:28.418006 (+ 57us) server_negotiation.cc:407] Connection header received
0620 18:42:28.418195 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:28.418201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:28.418262 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:28.418353 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:28.418949 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:28.419605 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:28.420659 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:28.421095 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:28.421219 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:28.423935 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:28.423939 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:28.423942 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:28.423967 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:28.426212 (+ 2245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:28.426778 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:28.426785 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:28.426788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:28.426898 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:28.427250 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:28.427257 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:28.427260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:28.427509 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:28.427645 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:28.427958 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 18:42:28.428246 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":137,"server-negotiator.queue_time_us":45}
I20260620 18:42:28.428573 12882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:28.417776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:45955 (local address 127.0.0.1:60398)
0620 18:42:28.417942 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:28.417961 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:28.418056 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:28.418366 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:28.418370 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:28.418704 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:28.418830 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:28.418837 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:28.419722 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:28.419729 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:28.420534 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:28.420543 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:28.421227 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:28.421232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:28.421433 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:28.421881 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:28.421897 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:28.423763 (+ 1866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:28.426333 (+ 2570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:28.426341 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:28.426345 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:28.426671 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:28.427003 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:28.427007 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:28.427011 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:28.427146 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:28.427624 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:28.427629 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:28.427748 (+ 119us) client_negotiation.cc:769] Sending connection context
0620 18:42:28.427890 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 18:42:28.428132 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":4}
I20260620 18:42:28.433359 12878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d: Bootstrap starting.
I20260620 18:42:28.436446 12878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:28.437439 12878 log.cc:824] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:28.439921 12878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d: No bootstrap required, opened a new log
I20260620 18:42:28.446625 12878 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 37095 } } peers { permanent_uuid: "c2efcc70af1e4061b9d972d5b56de43d" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 45955 } }
I20260620 18:42:28.446900 12878 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:28.446985 12878 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2efcc70af1e4061b9d972d5b56de43d, State: Initialized, Role: FOLLOWER
I20260620 18:42:28.447449 12878 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 37095 } } peers { permanent_uuid: "c2efcc70af1e4061b9d972d5b56de43d" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 45955 } }
I20260620 18:42:28.448154 12884 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 37095 } } peers { permanent_uuid: "c2efcc70af1e4061b9d972d5b56de43d" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 45955 } } }
I20260620 18:42:28.448467 12884 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:42:28.449097 12878 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:42:28.456503 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 12224
W20260620 18:42:28.457830 12895 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:42:28.457962 12895 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:42:28.462517 12724 master_runner.cc:428] Detected that this master c2efcc70af1e4061b9d972d5b56de43d is joining an existing cluster
I20260620 18:42:28.462641 12724 master_runner.cc:432] Initiating AddMaster RPC to add 127.5.225.188:45955
Jun 20 18:42:28 dist-test-slave-k6lg krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.225.129@KRBTEST.COM
I20260620 18:42:28.479789 12897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:28.457893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59806 (local address 127.5.225.129:42219)
0620 18:42:28.458579 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:28.458588 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:28.458617 (+ 29us) server_negotiation.cc:407] Connection header received
0620 18:42:28.458689 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:28.458698 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:28.458947 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:28.460833 (+ 1886us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:28.461057 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:28.462644 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:28.464811 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:28.465656 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:28.465791 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:28.471675 (+ 5884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:28.471689 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:28.471705 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:28.471760 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:28.475642 (+ 3882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:28.476504 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:28.476513 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:28.476516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:28.476669 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:28.477156 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:28.477163 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:28.477166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:28.477584 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:28.477875 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:28.479099 (+ 1224us) server_negotiation.cc:299] Negotiation successful
0620 18:42:28.479359 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":640,"threads_started":1}
I20260620 18:42:28.481312 12347 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:28.486410 12695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:28.468714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59982 (local address 127.5.225.190:34625)
0620 18:42:28.468896 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:28.468902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:28.468919 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:42:28.468975 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:28.468980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:28.469055 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:28.469153 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:28.470737 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:28.472080 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:28.474684 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:28.475557 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:28.475671 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:28.478708 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:28.478714 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:28.478721 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:28.478772 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:28.483185 (+ 4413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:28.483979 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:28.483985 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:28.483988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:28.484151 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:28.484629 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:28.484635 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:28.484638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:28.485005 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:28.485148 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:28.485797 (+ 649us) server_negotiation.cc:299] Negotiation successful
0620 18:42:28.486020 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 18:42:28.487843 12902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:28.467669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:34625 (local address 127.0.0.1:59982)
0620 18:42:28.468313 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:28.468331 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:28.468463 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:28.469343 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:28.469350 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:28.470057 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:28.470542 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:28.470553 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:28.472290 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:28.472298 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:28.474493 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:28.474511 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:28.475994 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:28.476003 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:28.476265 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:28.476982 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:28.477017 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:28.478468 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:28.483337 (+ 4869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:28.483343 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:28.483347 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:28.483829 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:28.484298 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:28.484303 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:28.484306 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:28.484499 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:28.485227 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:28.485231 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:28.485449 (+ 218us) client_negotiation.cc:769] Sending connection context
0620 18:42:28.486832 (+ 1383us) client_negotiation.cc:240] Negotiation successful
0620 18:42:28.487189 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":1408,"threads_started":1}
I20260620 18:42:28.503036 12811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:28.468907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47386 (local address 127.5.225.189:37095)
0620 18:42:28.469069 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:28.469074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:28.488549 (+ 19475us) server_negotiation.cc:407] Connection header received
0620 18:42:28.488799 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:28.488806 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:28.488887 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:28.488959 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:28.490077 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:28.491234 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:28.493157 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:28.493837 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:28.493934 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:28.496353 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:28.496360 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:28.496365 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:28.496402 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:28.499793 (+ 3391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:28.500741 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:28.500747 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:28.500750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:28.500895 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:28.501324 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:28.501329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:28.501333 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:28.501677 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:28.501813 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:28.502463 (+ 650us) server_negotiation.cc:299] Negotiation successful
0620 18:42:28.502691 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:42:28.504282 12902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:28.468305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:37095 (local address 127.0.0.1:47386)
0620 18:42:28.488420 (+ 20115us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:28.488440 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:28.488643 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:28.489160 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:28.489171 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:28.489663 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:28.489910 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:28.489921 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:28.491421 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:28.491426 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:28.492993 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:28.493008 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:28.494202 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:28.494210 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:28.494394 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:28.495041 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:28.495067 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:28.496155 (+ 1088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:28.500000 (+ 3845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:28.500006 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:28.500010 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:28.500585 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:28.501046 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:28.501051 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:28.501054 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:28.501200 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:28.501909 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:28.501914 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:28.502117 (+ 203us) client_negotiation.cc:769] Sending connection context
0620 18:42:28.503564 (+ 1447us) client_negotiation.cc:240] Negotiation successful
0620 18:42:28.503875 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19937,"threads_started":1}
Jun 20 18:42:28 dist-test-slave-k6lg krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980945, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.225.190@KRBTEST.COM for oryx/127.5.225.188@KRBTEST.COM
I20260620 18:42:28.520834 12881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:28.469817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60402 (local address 127.5.225.188:45955)
0620 18:42:28.469960 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:28.469965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:28.499560 (+ 29595us) server_negotiation.cc:407] Connection header received
0620 18:42:28.502669 (+ 3109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:28.502684 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:28.502753 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:28.502816 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:28.504952 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:28.506111 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:28.510861 (+ 4750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:28.511581 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:28.511673 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:28.514047 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:28.514053 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:28.514058 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:28.514088 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:28.517979 (+ 3891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:28.518748 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:28.518753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:28.518757 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:28.518889 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:28.519224 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:28.519231 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:28.519235 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:28.519586 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:28.519717 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:28.520150 (+ 433us) server_negotiation.cc:299] Negotiation successful
0620 18:42:28.520384 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 18:42:28.521576 12903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:28.469483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:45955 (local address 127.0.0.1:60402)
0620 18:42:28.499472 (+ 29989us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:28.499491 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:28.502481 (+ 2990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:28.503052 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:28.503059 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:28.503765 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:28.504067 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:28.504078 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:28.506481 (+ 2403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:28.506487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:28.510722 (+ 4235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:28.510735 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:28.511939 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:28.511945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:28.512104 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:28.512691 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:28.512727 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:28.513896 (+ 1169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:28.518107 (+ 4211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:28.518112 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:28.518116 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:28.518628 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:28.519001 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:28.519006 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:28.519013 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:28.519136 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:28.519778 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:28.519782 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:28.519911 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 18:42:28.520283 (+ 372us) client_negotiation.cc:240] Negotiation successful
0620 18:42:28.520580 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29853,"thread_start_us":30309,"threads_started":1}
I20260620 18:42:28.525456 12718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:28.507274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:45955 (local address 127.0.0.1:60408)
0620 18:42:28.507440 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:28.507458 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:28.507553 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:28.508682 (+ 1129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:28.508688 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:28.509164 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:28.509377 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:28.509389 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:28.511403 (+ 2014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:28.511411 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:28.512796 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:28.512811 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:28.513897 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:28.513905 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:28.514079 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:28.514676 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:28.514702 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:28.517534 (+ 2832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:28.522819 (+ 5285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:28.522824 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:28.522828 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:28.523303 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:28.523747 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:28.523752 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:28.523756 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:28.523904 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:28.524525 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:28.524530 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:28.524696 (+ 166us) client_negotiation.cc:769] Sending connection context
0620 18:42:28.524835 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 18:42:28.525095 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 18:42:28.526763 12906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:28.507770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60408 (local address 127.5.225.188:45955)
0620 18:42:28.508318 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:28.508323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:28.508344 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:42:28.508429 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:28.508434 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:28.508511 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:28.509550 (+ 1039us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:28.509897 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:28.511250 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:28.512974 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:28.513686 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:28.514373 (+ 687us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:28.517797 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:28.517805 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:28.517809 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:28.517850 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:28.522661 (+ 4811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:28.523455 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:28.523461 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:28.523466 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:28.523603 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:28.524046 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:28.524050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:28.524054 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:28.524384 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:28.526040 (+ 1656us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:28.526149 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 18:42:28.526384 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":270,"threads_started":1}
I20260620 18:42:28.528164 12163 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.5.225.188:45955: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c2efcc70af1e4061b9d972d5b56de43d"
member_type: NON_VOTER
last_known_addr {
host: "127.5.225.188"
port: 45955
}
attrs {
promote: true
}
}
dest_uuid: "6cf69d27a27142a9b0c07415bb23d249"
cas_config_opid_index: 5
I20260620 18:42:28.529166 12163 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: NON_VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: true } } peers { permanent_uuid: "c2efcc70af1e4061b9d972d5b56de43d" member_type: NON_VOTER last_known_addr { host: "127.5.225.188" port: 45955 } attrs { promote: true } }
I20260620 18:42:28.530782 12721 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c2efcc70af1e4061b9d972d5b56de43d (127.5.225.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: NON_VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: true } } peers { permanent_uuid: "c2efcc70af1e4061b9d972d5b56de43d" member_type: NON_VOTER last_known_addr { host: "127.5.225.188" port: 45955 } attrs { promote: true } } }
I20260620 18:42:28.534842 12771 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [term 1 LEARNER]: Refusing update from remote peer 6cf69d27a27142a9b0c07415bb23d249: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 18:42:28.535929 12721 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.5.225.188:45955
I20260620 18:42:28.536298 12853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:28.536464 12724 master.cc:537] Master@127.5.225.188:45955 shutting down...
I20260620 18:42:28.542609 12853 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [term 1 FOLLOWER]: Refusing update from remote peer 6cf69d27a27142a9b0c07415bb23d249: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 18:42:28.547173 12907 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: NON_VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 18:42:28.552099 12724 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:42:28.552629 12724 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:42:28.552943 12724 tablet_replica.cc:330] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d: stopping tablet replica
I20260620 18:42:28.555351 12721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6cf69d27a27142a9b0c07415bb23d249" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: NON_VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: true } } peers { permanent_uuid: "c2efcc70af1e4061b9d972d5b56de43d" member_type: NON_VOTER last_known_addr { host: "127.5.225.188" port: 45955 } attrs { promote: true } } }
I20260620 18:42:28.555651 12721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:28.563146 12908 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2efcc70af1e4061b9d972d5b56de43d" member_type: NON_VOTER last_known_addr { host: "127.5.225.188" port: 45955 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:42:28.570945 12724 master.cc:552] Master@127.5.225.188:45955 shutdown complete.
I20260620 18:42:28.571022 12724 master_runner.cc:305] Clearing existing system tablet
I20260620 18:42:28.570674 12771 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c2efcc70af1e4061b9d972d5b56de43d (127.5.225.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: NON_VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: true } } peers { permanent_uuid: "c2efcc70af1e4061b9d972d5b56de43d" member_type: NON_VOTER last_known_addr { host: "127.5.225.188" port: 45955 } attrs { promote: true } } }
I20260620 18:42:28.571455 12724 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:28.571553 12724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:28.573261 12718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:28.572814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:45955 (local address 127.0.0.1:60412)
0620 18:42:28.572972 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:28.573099 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 18:42:28.575423 12799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6cf69d27a27142a9b0c07415bb23d249" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: NON_VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: true } } peers { permanent_uuid: "c2efcc70af1e4061b9d972d5b56de43d" member_type: NON_VOTER last_known_addr { host: "127.5.225.188" port: 45955 } attrs { promote: true } } }
I20260620 18:42:28.575775 12799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [sys.catalog]: This master's current role is: LEARNER
I20260620 18:42:28.576890 12909 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249: attempting to promote NON_VOTER 45a9ce1454f24d488dd7295235c75786 to VOTER
I20260620 18:42:28.577529 12909 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: false } } peers { permanent_uuid: "c2efcc70af1e4061b9d972d5b56de43d" member_type: NON_VOTER last_known_addr { host: "127.5.225.188" port: 45955 } attrs { promote: true } }
I20260620 18:42:28.579741 12771 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [term 1 LEARNER]: Refusing update from remote peer 6cf69d27a27142a9b0c07415bb23d249: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 18:42:28.580224 12909 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 18:42:28.581449 12724 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.001s sys 0.008s
I20260620 18:42:28.582659 12721 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 45a9ce1454f24d488dd7295235c75786 (127.5.225.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: false } } peers { permanent_uuid: "c2efcc70af1e4061b9d972d5b56de43d" member_type: NON_VOTER last_known_addr { host: "127.5.225.188" port: 45955 } attrs { promote: true } } }
I20260620 18:42:28.587146 12770 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 45a9ce1454f24d488dd7295235c75786 (127.5.225.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: false } } peers { permanent_uuid: "c2efcc70af1e4061b9d972d5b56de43d" member_type: NON_VOTER last_known_addr { host: "127.5.225.188" port: 45955 } attrs { promote: true } } }
I20260620 18:42:28.591183 12799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6cf69d27a27142a9b0c07415bb23d249" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: false } } peers { permanent_uuid: "c2efcc70af1e4061b9d972d5b56de43d" member_type: NON_VOTER last_known_addr { host: "127.5.225.188" port: 45955 } attrs { promote: true } } }
I20260620 18:42:28.591416 12799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:42:28.592923 12718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:28.592297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:45955 (local address 127.0.0.1:60420)
0620 18:42:28.592664 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:28.592758 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260}
I20260620 18:42:28.594452 12718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:28.593958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:45955 (local address 127.0.0.1:60422)
0620 18:42:28.594144 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:28.594258 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 18:42:28.594763 12149 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 -> Peer c2efcc70af1e4061b9d972d5b56de43d (127.5.225.188:45955): Couldn't send request to peer c2efcc70af1e4061b9d972d5b56de43d. Status: Network error: Client connection negotiation failed: client connection to 127.5.225.188:45955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 18:42:28.598289 12907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6cf69d27a27142a9b0c07415bb23d249" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: false } } peers { permanent_uuid: "c2efcc70af1e4061b9d972d5b56de43d" member_type: NON_VOTER last_known_addr { host: "127.5.225.188" port: 45955 } attrs { promote: true } } }
I20260620 18:42:28.598610 12907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:28.598866 12907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6cf69d27a27142a9b0c07415bb23d249" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: false } } peers { permanent_uuid: "c2efcc70af1e4061b9d972d5b56de43d" member_type: NON_VOTER last_known_addr { host: "127.5.225.188" port: 45955 } attrs { promote: true } } }
I20260620 18:42:28.599093 12907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:28.599596 12914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:28.599924 12724 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:28.600205 12724 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.007s sys 0.009s
I20260620 18:42:28.600328 12724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-2/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-2/wal
uuid: "c2efcc70af1e4061b9d972d5b56de43d"
format_stamp: "Formatted at 2026-06-20 18:42:28 on dist-test-slave-k6lg"
server_key: "0fce1fee7760eb5a11a4206a02f99c08"
server_key_iv: "e261b20ccf25834217bcd971f89a3377"
server_key_version: "encryptionkey@0"
I20260620 18:42:28.605412 12724 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:42:28.612293 12724 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 18:42:28.612454 12724 log.cc:1196] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 18:42:28.612898 12724 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d: Deleting consensus metadata
I20260620 18:42:28.613143 12724 master_runner.cc:315] Copying system tablet from 127.5.225.190:34625
I20260620 18:42:28.619711 12724 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d: tablet copy: Beginning tablet copy session from remote peer at address 127.5.225.190:34625
I20260620 18:42:28.639628 12921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:28.620465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:34625 (local address 127.0.0.1:59986)
0620 18:42:28.625482 (+ 5017us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:28.625508 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:28.625672 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:28.626056 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:28.626061 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:28.626612 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:28.626875 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:28.626887 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:28.628249 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:28.628254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:28.629804 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:28.629819 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:28.630787 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:28.630794 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:28.630990 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:28.631701 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:28.631728 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:28.632770 (+ 1042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:28.636518 (+ 3748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:28.636523 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:28.636528 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:28.636987 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:28.637393 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:28.637398 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:28.637402 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:28.637569 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:28.638451 (+ 882us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:28.638456 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:28.638649 (+ 193us) client_negotiation.cc:769] Sending connection context
0620 18:42:28.638782 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 18:42:28.639098 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4877,"thread_start_us":4757,"threads_started":1}
I20260620 18:42:28.640601 12695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:28.620624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59986 (local address 127.5.225.190:34625)
0620 18:42:28.620862 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:28.620869 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:28.625626 (+ 4757us) server_negotiation.cc:407] Connection header received
0620 18:42:28.625827 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:28.625833 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:28.625915 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:28.626013 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:28.627038 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:28.628105 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:28.629958 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:28.630597 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:28.630712 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:28.632986 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:28.632992 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:28.632997 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:28.633037 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:28.636389 (+ 3352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:28.637134 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:28.637139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:28.637142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:28.637286 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:28.637732 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:28.637737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:28.637740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:28.638076 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:28.639171 (+ 1095us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:28.639327 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 18:42:28.640096 (+ 769us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260620 18:42:28.641244 12183 tablet_copy_service.cc:140] P 6cf69d27a27142a9b0c07415bb23d249: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c2efcc70af1e4061b9d972d5b56de43d ({username='oryx', principal='oryx/127.5.225.188@KRBTEST.COM'} at 127.0.0.1:59986)
I20260620 18:42:28.641419 12183 tablet_copy_service.cc:161] P 6cf69d27a27142a9b0c07415bb23d249: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c2efcc70af1e4061b9d972d5b56de43d at {username='oryx', principal='oryx/127.5.225.188@KRBTEST.COM'} at 127.0.0.1:59986: session id = c2efcc70af1e4061b9d972d5b56de43d-00000000000000000000000000000000
I20260620 18:42:28.643677 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 12224
I20260620 18:42:28.645076 12183 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 18:42:28.647078 12724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 18:42:28.654948 12724 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d: tablet copy: Starting download of 0 data blocks...
I20260620 18:42:28.655252 12724 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d: tablet copy: Starting download of 1 WAL segments...
I20260620 18:42:28.659528 12724 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 18:42:28.663880 12183 tablet_copy_service.cc:342] P 6cf69d27a27142a9b0c07415bb23d249: Request end of tablet copy session c2efcc70af1e4061b9d972d5b56de43d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.5.225.188@KRBTEST.COM'} at 127.0.0.1:59986
I20260620 18:42:28.664019 12183 tablet_copy_service.cc:433] P 6cf69d27a27142a9b0c07415bb23d249: ending tablet copy session c2efcc70af1e4061b9d972d5b56de43d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c2efcc70af1e4061b9d972d5b56de43d
I20260620 18:42:28.664769 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.129:42219
--local_ip_for_outbound_sockets=127.5.225.129
--tserver_master_addrs=127.5.225.190:34625
--webserver_port=39769
--webserver_interface=127.5.225.129
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.129.keytab
--principal=oryx/127.5.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:42967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.225.190:34625,127.5.225.189:37095
--tserver_master_addrs=127.5.225.190:34625,127.5.225.189:37095,127.5.225.188:45955 with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:42:28.674650 12724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:28.677453 12923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:28.683658 12924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:28.700229 12926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:28.701081 12724 server_base.cc:1034] running on GCE node
I20260620 18:42:28.701447 12724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:28.702100 12724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:28.703322 12724 hybrid_clock.cc:648] HybridClock initialized: now 1781980948703300 us; error 50 us; skew 500 ppm
I20260620 18:42:28.703588 12724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:28 dist-test-slave-k6lg krb5kdc[12118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980948, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.225.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:28.705673 12724 init.cc:377] Logged in from keytab as oryx/127.5.225.188@KRBTEST.COM (short username oryx)
I20260620 18:42:28.709837 12724 webserver.cc:460] Webserver started at http://127.5.225.188:46259/ using document root <none> and password file <none>
I20260620 18:42:28.710201 12724 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:28.710304 12724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:28.713326 12724 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 18:42:28.718607 12933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:28.719072 12724 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:42:28.719189 12724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-2/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-2/wal
uuid: "c2efcc70af1e4061b9d972d5b56de43d"
format_stamp: "Formatted at 2026-06-20 18:42:28 on dist-test-slave-k6lg"
server_key: "0fce1fee7760eb5a11a4206a02f99c08"
server_key_iv: "e261b20ccf25834217bcd971f89a3377"
server_key_version: "encryptionkey@0"
I20260620 18:42:28.719383 12724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:28.733470 12724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:28.737532 12724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:28.756656 12724 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.188:45955
I20260620 18:42:28.756728 12984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.188:45955 every 8 connection(s)
I20260620 18:42:28.757730 12724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/master-2/data/info.pb
I20260620 18:42:28.763917 12985 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:42:28.767771 12985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d: Bootstrap starting.
I20260620 18:42:28.786240 12985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:42:28.786738 12985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d: Bootstrap complete.
I20260620 18:42:28.787834 12985 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: false } } peers { permanent_uuid: "c2efcc70af1e4061b9d972d5b56de43d" member_type: NON_VOTER last_known_addr { host: "127.5.225.188" port: 45955 } attrs { promote: true } }
I20260620 18:42:28.788152 12985 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c2efcc70af1e4061b9d972d5b56de43d, State: Initialized, Role: LEARNER
I20260620 18:42:28.788478 12985 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: false } } peers { permanent_uuid: "c2efcc70af1e4061b9d972d5b56de43d" member_type: NON_VOTER last_known_addr { host: "127.5.225.188" port: 45955 } attrs { promote: true } }
I20260620 18:42:28.789407 12988 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: false } } peers { permanent_uuid: "c2efcc70af1e4061b9d972d5b56de43d" member_type: NON_VOTER last_known_addr { host: "127.5.225.188" port: 45955 } attrs { promote: true } } }
I20260620 18:42:28.789791 12988 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [sys.catalog]: This master's current role is: LEARNER
I20260620 18:42:28.789848 12985 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:42:28.799327 12999 catalog_manager.cc:1259] Loaded cluster ID: fdd42e8c2ff34c219f495835c8ada190
I20260620 18:42:28.799428 12999 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d: loading cluster ID for follower catalog manager: success
I20260620 18:42:28.802754 12999 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d: acquiring CA information for follower catalog manager: success
I20260620 18:42:28.804962 12999 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 18:42:29.034132 12922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:29.034646 12922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:29.034785 12922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:29.042471 12922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:29.042619 12922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:29.042835 12922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.129
I20260620 18:42:29.052630 12922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:42967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.129.keytab
--principal=oryx/127.5.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.129:42219
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.225.129
--webserver_port=39769
--tserver_master_addrs=127.5.225.190:34625,127.5.225.189:37095,127.5.225.188:45955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.129
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:29.053930 12922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:29.055296 12922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:29.068157 13004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:29.068300 13005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:29.068267 13007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:29.069339 12922 server_base.cc:1034] running on GCE node
I20260620 18:42:29.069962 12922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:29.071020 12922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:29.072252 12922 hybrid_clock.cc:648] HybridClock initialized: now 1781980949072210 us; error 53 us; skew 500 ppm
I20260620 18:42:29.072557 12922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:29 dist-test-slave-k6lg krb5kdc[12118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980949, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:29.075587 12922 init.cc:377] Logged in from keytab as oryx/127.5.225.129@KRBTEST.COM (short username oryx)
I20260620 18:42:29.076967 12922 webserver.cc:460] Webserver started at http://127.5.225.129:39769/ using document root <none> and password file <none>
I20260620 18:42:29.077618 12922 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:29.077733 12922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:29.083714 12922 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 18:42:29.087271 13014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:29.089648 12922 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 18:42:29.089856 12922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-0/wal
uuid: "911d8be242834719ac569e965601a2e2"
format_stamp: "Formatted at 2026-06-20 18:42:26 on dist-test-slave-k6lg"
server_key: "911e27054016cb2af5f3e1c40a37c3cf"
server_key_iv: "cb517d62c1f74345c672fefebc338be0"
server_key_version: "encryptionkey@0"
I20260620 18:42:29.091018 12922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:29.094624 12718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:29.087959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:45955 (local address 127.0.0.1:60432)
0620 18:42:29.088142 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:29.088163 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:29.088302 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:29.089128 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:29.089136 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:29.089172 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:29.089420 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:29.089435 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:29.091390 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:29.091395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:29.092666 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:29.092685 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:29.093795 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:29.093801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:29.094035 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:29.094069 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 18:42:29.094153 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 18:42:29.094326 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 18:42:29.094733 13015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:29.088105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60432 (local address 127.5.225.188:45955)
0620 18:42:29.088726 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:29.088733 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:29.088755 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:42:29.088831 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:29.088839 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:29.088939 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:29.089603 (+ 664us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:29.089855 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:29.091198 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:29.092813 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:29.093623 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:29.093770 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:29.094232 (+ 462us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:29.094282 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 18:42:29.094407 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":290,"threads_started":1}
I20260620 18:42:29.095497 12960 raft_consensus.cc:1203] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 18:42:29.096211 12988 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cf69d27a27142a9b0c07415bb23d249. Latest consensus state: current_term: 1 leader_uuid: "6cf69d27a27142a9b0c07415bb23d249" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: false } } peers { permanent_uuid: "c2efcc70af1e4061b9d972d5b56de43d" member_type: NON_VOTER last_known_addr { host: "127.5.225.188" port: 45955 } attrs { promote: true } } }
I20260620 18:42:29.096511 12988 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [sys.catalog]: This master's current role is: LEARNER
I20260620 18:42:29.123994 12922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:29.127919 12922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:29.128185 12922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:29.129149 12922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:29.130512 12922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:29.130626 12922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:29.130750 12922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:29.130834 12922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:29.162065 12922 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.129:42219
I20260620 18:42:29.162146 13127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.129:42219 every 8 connection(s)
I20260620 18:42:29.163470 12922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
I20260620 18:42:29.168561 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 12922
I20260620 18:42:29.168915 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 12358
Jun 20 18:42:29 dist-test-slave-k6lg krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980949, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.225.129@KRBTEST.COM for oryx/127.5.225.189@KRBTEST.COM
Jun 20 18:42:29 dist-test-slave-k6lg krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980949, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.225.129@KRBTEST.COM for oryx/127.5.225.188@KRBTEST.COM
Jun 20 18:42:29 dist-test-slave-k6lg krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980949, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.225.129@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM
Jun 20 18:42:29 dist-test-slave-k6lg krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.225.130@KRBTEST.COM
I20260620 18:42:29.196180 13015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:29.166499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:49141 (local address 127.5.225.188:45955)
0620 18:42:29.166724 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:29.166733 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:29.169093 (+ 2360us) server_negotiation.cc:407] Connection header received
0620 18:42:29.175616 (+ 6523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:29.175624 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:29.175733 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:29.175816 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:29.178666 (+ 2850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:29.179826 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:29.182603 (+ 2777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:29.183256 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:29.183345 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:29.189261 (+ 5916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:29.189267 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:29.189271 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:29.189310 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:29.192822 (+ 3512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:29.193714 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:29.193720 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:29.193723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:29.193869 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:29.194313 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:29.194318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:29.194323 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:29.194739 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:29.194857 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:29.195602 (+ 745us) server_negotiation.cc:299] Negotiation successful
0620 18:42:29.195853 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 18:42:29.196413 13132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:29.166362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:41785 (local address 127.5.225.189:37095)
0620 18:42:29.174609 (+ 8247us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:29.174619 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:29.174661 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:42:29.174742 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:29.174751 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:29.174891 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:29.175035 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:29.178392 (+ 3357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:29.179494 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:29.182615 (+ 3121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:29.183235 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:29.183325 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:29.188353 (+ 5028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:29.188360 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:29.188365 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:29.188406 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:29.192285 (+ 3879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:29.193239 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:29.193245 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:29.193249 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:29.193395 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:29.194001 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:29.194006 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:29.194010 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:29.194435 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:29.194604 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:29.195657 (+ 1053us) server_negotiation.cc:299] Negotiation successful
0620 18:42:29.196015 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8010,"thread_start_us":531,"threads_started":1}
I20260620 18:42:29.197503 13137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:29.170301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:37005 (local address 127.5.225.190:34625)
0620 18:42:29.178015 (+ 7714us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:29.178021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:29.178547 (+ 526us) server_negotiation.cc:407] Connection header received
0620 18:42:29.178623 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:29.178629 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:29.178704 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:29.178806 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:29.180034 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:29.181145 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:29.183197 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:29.183914 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:29.184113 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:29.189171 (+ 5058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:29.189181 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:29.189187 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:29.189271 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:29.193149 (+ 3878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:29.194067 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:29.194075 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:29.194081 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:29.194244 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:29.194699 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:29.194704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:29.194707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:29.195184 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:29.195340 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:29.196932 (+ 1592us) server_negotiation.cc:299] Negotiation successful
0620 18:42:29.197142 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7561,"thread_start_us":7453,"threads_started":1}
I20260620 18:42:29.198258 13134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:29.167106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:45955 (local address 127.5.225.129:49141)
0620 18:42:29.168977 (+ 1871us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:29.168998 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:29.175324 (+ 6326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:29.176077 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:29.176084 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:29.176866 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:29.177790 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:29.177814 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:29.180016 (+ 2202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:29.180020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:29.182446 (+ 2426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:29.182461 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:29.183673 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:29.183680 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:29.184131 (+ 451us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:29.184889 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:29.184917 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:29.189038 (+ 4121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:29.192996 (+ 3958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:29.193003 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:29.193007 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:29.193567 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:29.194012 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:29.194019 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:29.194023 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:29.194191 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:29.194931 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:29.194935 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:29.195184 (+ 249us) client_negotiation.cc:769] Sending connection context
0620 18:42:29.196496 (+ 1312us) client_negotiation.cc:240] Negotiation successful
0620 18:42:29.196774 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1694,"mutex_wait_us":332,"thread_start_us":2033,"threads_started":1}
I20260620 18:42:29.198282 13133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:29.167106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:37095 (local address 127.5.225.129:41785)
0620 18:42:29.168071 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:29.168123 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:29.170594 (+ 2471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:29.175635 (+ 5041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:29.175653 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:29.176731 (+ 1078us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:29.178152 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:29.178200 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:29.179720 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:29.179726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:29.182452 (+ 2726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:29.182467 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:29.183672 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:29.183680 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:29.184039 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:29.184731 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:29.184763 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:29.188007 (+ 3244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:29.192443 (+ 4436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:29.192454 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:29.192473 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:29.193086 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:29.193548 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:29.193552 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:29.193555 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:29.193865 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:29.194751 (+ 886us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:29.194760 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:29.195193 (+ 433us) client_negotiation.cc:769] Sending connection context
0620 18:42:29.196664 (+ 1471us) client_negotiation.cc:240] Negotiation successful
0620 18:42:29.196929 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"mutex_wait_us":880,"spinlock_wait_cycles":34048,"thread_start_us":387,"threads_started":1}
I20260620 18:42:29.199937 13136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:29.169849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:34625 (local address 127.5.225.129:37005)
0620 18:42:29.178280 (+ 8431us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:29.178302 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:29.178435 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:29.178991 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:29.178996 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:29.179479 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:29.179862 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:29.179876 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:29.181303 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:29.181308 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:29.183035 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:29.183048 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:29.184842 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:29.184858 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:29.185083 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:29.185737 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:29.185765 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:29.188937 (+ 3172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:29.193309 (+ 4372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:29.193316 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:29.193320 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:29.193910 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:29.194377 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:29.194383 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:29.194389 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:29.194574 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:29.195396 (+ 822us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:29.195400 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:29.195577 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 18:42:29.195724 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 18:42:29.197740 (+ 2016us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8237,"mutex_wait_us":117,"thread_start_us":8063,"threads_started":1}
I20260620 18:42:29.200129 13130 heartbeater.cc:347] Connected to a master server at 127.5.225.189:37095
I20260620 18:42:29.200152 13129 heartbeater.cc:347] Connected to a master server at 127.5.225.188:45955
I20260620 18:42:29.200495 13130 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:29.200500 13129 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:29.200937 13128 heartbeater.cc:347] Connected to a master server at 127.5.225.190:34625
I20260620 18:42:29.201071 13128 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:29.201567 13129 heartbeater.cc:511] Master 127.5.225.188:45955 requested a full tablet report, sending...
I20260620 18:42:29.201601 13128 heartbeater.cc:511] Master 127.5.225.190:34625 requested a full tablet report, sending...
I20260620 18:42:29.202358 13130 heartbeater.cc:511] Master 127.5.225.189:37095 requested a full tablet report, sending...
I20260620 18:42:29.203166 13138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:29.170602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56122 (local address 127.5.225.130:45529)
0620 18:42:29.177777 (+ 7175us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:29.177787 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:29.178925 (+ 1138us) server_negotiation.cc:407] Connection header received
0620 18:42:29.178996 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:29.179004 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:29.179275 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:29.179413 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:29.181672 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:29.183342 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:29.185161 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:29.186013 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:29.186142 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:29.193612 (+ 7470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:29.193627 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:29.193645 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:29.193689 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:29.197567 (+ 3878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:29.198461 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:29.198471 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:29.198476 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:29.198630 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:29.199066 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:29.199071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:29.199075 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:29.199504 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:29.199782 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:29.202513 (+ 2731us) server_negotiation.cc:299] Negotiation successful
0620 18:42:29.202794 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7008,"thread_start_us":6837,"threads_started":1}
I20260620 18:42:29.203476 12163 ts_manager.cc:194] Re-registered known tserver with Master: 911d8be242834719ac569e965601a2e2 (127.5.225.129:42219)
I20260620 18:42:29.204027 12761 ts_manager.cc:194] Registered new tserver with Master: 911d8be242834719ac569e965601a2e2 (127.5.225.129:42219)
I20260620 18:42:29.204324 12481 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:29.204479 12163 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.225.129@KRBTEST.COM'} at 127.5.225.129:37005
I20260620 18:42:29.205543 12950 ts_manager.cc:194] Registered new tserver with Master: 911d8be242834719ac569e965601a2e2 (127.5.225.129:42219)
I20260620 18:42:29.289229 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 12358
I20260620 18:42:29.307384 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.130:45529
--local_ip_for_outbound_sockets=127.5.225.130
--tserver_master_addrs=127.5.225.190:34625
--webserver_port=35899
--webserver_interface=127.5.225.130
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.130.keytab
--principal=oryx/127.5.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:42967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.225.190:34625,127.5.225.189:37095
--tserver_master_addrs=127.5.225.190:34625,127.5.225.189:37095,127.5.225.188:45955 with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:29.633992 13140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:29.634277 13140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:29.634455 13140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:29.641624 13140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:29.641763 13140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:29.641932 13140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.130
I20260620 18:42:29.651523 13140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:42967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.130.keytab
--principal=oryx/127.5.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.130:45529
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.225.130
--webserver_port=35899
--tserver_master_addrs=127.5.225.190:34625,127.5.225.189:37095,127.5.225.188:45955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.130
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:29.652817 13140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:29.654103 13140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:29.670493 13145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:29.670594 13146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:29.670845 13148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:29.672719 13140 server_base.cc:1034] running on GCE node
I20260620 18:42:29.673363 13140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:29.674386 13140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:29.675614 13140 hybrid_clock.cc:648] HybridClock initialized: now 1781980949675541 us; error 55 us; skew 500 ppm
I20260620 18:42:29.675947 13140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:29 dist-test-slave-k6lg krb5kdc[12118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980949, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:29.679737 13140 init.cc:377] Logged in from keytab as oryx/127.5.225.130@KRBTEST.COM (short username oryx)
I20260620 18:42:29.681938 13140 webserver.cc:460] Webserver started at http://127.5.225.130:35899/ using document root <none> and password file <none>
I20260620 18:42:29.682744 13140 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:29.682857 13140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:29.688270 13140 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260620 18:42:29.691049 13155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:29.692382 13140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:42:29.692514 13140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-1/wal
uuid: "762e84cc86204351bc255b1e796e833f"
format_stamp: "Formatted at 2026-06-20 18:42:26 on dist-test-slave-k6lg"
server_key: "fff5fe99ed51ba2e3a388256c2ca97f4"
server_key_iv: "0d43efc4d3a2fe6df64c23c7460d05d4"
server_key_version: "encryptionkey@0"
I20260620 18:42:29.693431 13140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:29.705199 12721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6cf69d27a27142a9b0c07415bb23d249" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: false } } peers { permanent_uuid: "c2efcc70af1e4061b9d972d5b56de43d" member_type: NON_VOTER last_known_addr { host: "127.5.225.188" port: 45955 } attrs { promote: true } } }
I20260620 18:42:29.705474 12721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:29.706810 13160 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249: attempting to promote NON_VOTER c2efcc70af1e4061b9d972d5b56de43d to VOTER
I20260620 18:42:29.707429 13160 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: false } } peers { permanent_uuid: "c2efcc70af1e4061b9d972d5b56de43d" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 45955 } attrs { promote: false } }
I20260620 18:42:29.709338 12960 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [term 1 LEARNER]: Refusing update from remote peer 6cf69d27a27142a9b0c07415bb23d249: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 18:42:29.709385 12770 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [term 1 FOLLOWER]: Refusing update from remote peer 6cf69d27a27142a9b0c07415bb23d249: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 18:42:29.709867 13160 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2efcc70af1e4061b9d972d5b56de43d" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 45955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 18:42:29.710244 13161 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 18:42:29.712962 13161 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c2efcc70af1e4061b9d972d5b56de43d (127.5.225.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: false } } peers { permanent_uuid: "c2efcc70af1e4061b9d972d5b56de43d" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 45955 } attrs { promote: false } } }
I20260620 18:42:29.713508 12770 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c2efcc70af1e4061b9d972d5b56de43d (127.5.225.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: false } } peers { permanent_uuid: "c2efcc70af1e4061b9d972d5b56de43d" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 45955 } attrs { promote: false } } }
I20260620 18:42:29.717763 13160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6cf69d27a27142a9b0c07415bb23d249" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: false } } peers { permanent_uuid: "c2efcc70af1e4061b9d972d5b56de43d" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 45955 } attrs { promote: false } } }
I20260620 18:42:29.718052 13160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:29.717864 12721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6cf69d27a27142a9b0c07415bb23d249" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: false } } peers { permanent_uuid: "c2efcc70af1e4061b9d972d5b56de43d" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 45955 } attrs { promote: false } } }
I20260620 18:42:29.718247 12721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:29.719745 13166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6cf69d27a27142a9b0c07415bb23d249" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: false } } peers { permanent_uuid: "c2efcc70af1e4061b9d972d5b56de43d" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 45955 } attrs { promote: false } } }
I20260620 18:42:29.720058 13166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:42:29.720428 12721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6cf69d27a27142a9b0c07415bb23d249" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: false } } peers { permanent_uuid: "c2efcc70af1e4061b9d972d5b56de43d" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 45955 } attrs { promote: false } } }
I20260620 18:42:29.720705 12721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249 [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:29.721223 13140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:29.721124 12960 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c2efcc70af1e4061b9d972d5b56de43d (127.5.225.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: false } } peers { permanent_uuid: "c2efcc70af1e4061b9d972d5b56de43d" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 45955 } attrs { promote: false } } }
I20260620 18:42:29.724282 13167 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6cf69d27a27142a9b0c07415bb23d249" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: false } } peers { permanent_uuid: "c2efcc70af1e4061b9d972d5b56de43d" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 45955 } attrs { promote: false } } }
I20260620 18:42:29.724568 13167 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:42:29.725905 13140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:29.726207 13140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:29.727134 13140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:29.728432 13140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:29.728551 13140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:29.728669 13140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:29.728760 13140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:29.760805 13140 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.130:45529
I20260620 18:42:29.760838 13275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.130:45529 every 8 connection(s)
I20260620 18:42:29.762522 13140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
I20260620 18:42:29.767010 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 13140
I20260620 18:42:29.767314 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 12491
Jun 20 18:42:29 dist-test-slave-k6lg krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.225.131@KRBTEST.COM
Jun 20 18:42:29 dist-test-slave-k6lg krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980949, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.225.130@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM
Jun 20 18:42:29 dist-test-slave-k6lg krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980949, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.225.130@KRBTEST.COM for oryx/127.5.225.188@KRBTEST.COM
Jun 20 18:42:29 dist-test-slave-k6lg krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980949, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.225.130@KRBTEST.COM for oryx/127.5.225.189@KRBTEST.COM
I20260620 18:42:29.808739 13282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:29.768598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45616 (local address 127.5.225.131:42693)
0620 18:42:29.780880 (+ 12282us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:29.780893 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:29.781841 (+ 948us) server_negotiation.cc:407] Connection header received
0620 18:42:29.781938 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:29.781950 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:29.782304 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:29.782482 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:29.785350 (+ 2868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:29.787549 (+ 2199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:29.791256 (+ 3707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:29.792396 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:29.792562 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:29.801131 (+ 8569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:29.801145 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:29.801160 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:29.801206 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:29.805574 (+ 4368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:29.806441 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:29.806450 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:29.806453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:29.806615 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:29.807048 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:29.807054 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:29.807059 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:29.807450 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:29.807739 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:29.808085 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 18:42:29.808345 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12038,"thread_start_us":11862,"threads_started":1}
I20260620 18:42:29.810074 12614 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:29.814942 13279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:29.765146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:58975 (local address 127.5.225.188:45955)
0620 18:42:29.790612 (+ 25466us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:29.790621 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:29.790644 (+ 23us) server_negotiation.cc:407] Connection header received
0620 18:42:29.790726 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:29.790733 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:29.790826 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:29.790957 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:29.795612 (+ 4655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:29.796810 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:29.799741 (+ 2931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:29.800520 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:29.800649 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:29.806540 (+ 5891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:29.806549 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:29.806555 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:29.806601 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:29.810325 (+ 3724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:29.811289 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:29.811295 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:29.811299 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:29.811439 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:29.811957 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:29.811962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:29.811965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:29.812297 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:29.812439 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:29.814282 (+ 1843us) server_negotiation.cc:299] Negotiation successful
0620 18:42:29.814493 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25299,"thread_start_us":25086,"threads_started":1}
I20260620 18:42:29.815572 13283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:29.780474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:43595 (local address 127.5.225.190:34625)
0620 18:42:29.786743 (+ 6269us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:29.786749 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:29.788843 (+ 2094us) server_negotiation.cc:407] Connection header received
0620 18:42:29.789011 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:29.789018 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:29.789107 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:29.789217 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:29.797032 (+ 7815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:29.798241 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:29.800331 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:29.800858 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:29.800978 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:29.806597 (+ 5619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:29.806606 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:29.806613 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:29.806657 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:29.811681 (+ 5024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:29.812812 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:29.812818 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:29.812822 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:29.812971 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:29.813383 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:29.813387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:29.813390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:29.813719 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:29.813889 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:29.814981 (+ 1092us) server_negotiation.cc:299] Negotiation successful
0620 18:42:29.815212 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6105,"thread_start_us":6211,"threads_started":1}
I20260620 18:42:29.816248 13284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:29.786566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:34625 (local address 127.5.225.130:43595)
0620 18:42:29.788727 (+ 2161us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:29.788762 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:29.788901 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:29.795843 (+ 6942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:29.795848 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:29.796518 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:29.796868 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:29.796883 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:29.798341 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:29.798346 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:29.800187 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:29.800202 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:29.801613 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:29.801623 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:29.801826 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:29.802630 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:29.802644 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:29.806369 (+ 3725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:29.812133 (+ 5764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:29.812137 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:29.812140 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:29.812683 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:29.813116 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:29.813121 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:29.813124 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:29.813270 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:29.813941 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:29.813945 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:29.814099 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 18:42:29.814270 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 18:42:29.814502 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1789,"thread_start_us":1946,"threads_started":1}
I20260620 18:42:29.816269 13280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:29.765935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:45955 (local address 127.5.225.130:58975)
0620 18:42:29.775906 (+ 9971us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:29.775971 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:29.778875 (+ 2904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:29.793183 (+ 14308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:29.793201 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:29.794242 (+ 1041us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:29.795366 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:29.795401 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:29.796952 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:29.796960 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:29.799539 (+ 2579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:29.799556 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:29.800747 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:29.800754 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:29.801030 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:29.802292 (+ 1262us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:29.802334 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:29.806216 (+ 3882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:29.810505 (+ 4289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:29.810515 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:29.810530 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:29.811142 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:29.811574 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:29.811594 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:29.811599 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:29.811822 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:29.812510 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:29.812518 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:29.812927 (+ 409us) client_negotiation.cc:769] Sending connection context
0620 18:42:29.813223 (+ 296us) client_negotiation.cc:240] Negotiation successful
0620 18:42:29.813761 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9585,"thread_start_us":13238,"threads_started":1}
I20260620 18:42:29.817718 13276 heartbeater.cc:347] Connected to a master server at 127.5.225.190:34625
I20260620 18:42:29.817836 13277 heartbeater.cc:347] Connected to a master server at 127.5.225.188:45955
I20260620 18:42:29.818059 13276 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:29.818065 13277 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:29.819140 13276 heartbeater.cc:511] Master 127.5.225.190:34625 requested a full tablet report, sending...
I20260620 18:42:29.819289 13287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:29.790088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:37095 (local address 127.5.225.130:41805)
0620 18:42:29.797539 (+ 7451us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:29.797557 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:29.797716 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:29.798203 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:29.798216 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:29.799589 (+ 1373us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:29.799958 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:29.799972 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:29.801487 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:29.801492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:29.803406 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:29.803429 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:29.804686 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:29.804692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:29.804906 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:29.805459 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:29.805482 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:29.810529 (+ 5047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:29.815399 (+ 4870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:29.815407 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:29.815413 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:29.816040 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:29.816552 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:29.816560 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:29.816566 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:29.816759 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:29.817503 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:29.817507 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:29.817669 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 18:42:29.818263 (+ 594us) client_negotiation.cc:240] Negotiation successful
0620 18:42:29.818659 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7321,"thread_start_us":7133,"threads_started":1}
I20260620 18:42:29.820075 13277 heartbeater.cc:511] Master 127.5.225.188:45955 requested a full tablet report, sending...
I20260620 18:42:29.820514 13286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:29.790444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:41805 (local address 127.5.225.189:37095)
0620 18:42:29.790943 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:29.790952 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:29.797630 (+ 6678us) server_negotiation.cc:407] Connection header received
0620 18:42:29.797846 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:29.797851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:29.797923 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:29.797995 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:29.800168 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:29.801314 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:29.803616 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:29.804289 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:29.804383 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:29.810823 (+ 6440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:29.810832 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:29.810839 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:29.810893 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:29.815238 (+ 4345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:29.816208 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:29.816216 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:29.816221 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:29.816404 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:29.816907 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:29.816914 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:29.816920 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:29.817300 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:29.817439 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:29.819660 (+ 2221us) server_negotiation.cc:299] Negotiation successful
0620 18:42:29.819956 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"spinlock_wait_cycles":6016,"thread_start_us":346,"threads_started":1}
I20260620 18:42:29.820960 12163 ts_manager.cc:194] Re-registered known tserver with Master: 762e84cc86204351bc255b1e796e833f (127.5.225.130:45529)
I20260620 18:42:29.821204 12950 ts_manager.cc:194] Registered new tserver with Master: 762e84cc86204351bc255b1e796e833f (127.5.225.130:45529)
I20260620 18:42:29.822036 12163 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.225.130@KRBTEST.COM'} at 127.5.225.130:43595
I20260620 18:42:29.822234 13278 heartbeater.cc:347] Connected to a master server at 127.5.225.189:37095
I20260620 18:42:29.822388 13278 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:29.822939 13278 heartbeater.cc:511] Master 127.5.225.189:37095 requested a full tablet report, sending...
I20260620 18:42:29.824075 12761 ts_manager.cc:194] Registered new tserver with Master: 762e84cc86204351bc255b1e796e833f (127.5.225.130:45529)
I20260620 18:42:29.906499 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 12491
I20260620 18:42:29.922442 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.131:42693
--local_ip_for_outbound_sockets=127.5.225.131
--tserver_master_addrs=127.5.225.190:34625
--webserver_port=40319
--webserver_interface=127.5.225.131
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.131.keytab
--principal=oryx/127.5.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:42967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.225.190:34625,127.5.225.189:37095
--tserver_master_addrs=127.5.225.190:34625,127.5.225.189:37095,127.5.225.188:45955 with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:42:30.208204 13128 heartbeater.cc:503] Master 127.5.225.190:34625 was elected leader, sending a full tablet report...
W20260620 18:42:30.250620 13289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:30.250946 13289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:30.251111 13289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:30.258894 13289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:30.259047 13289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:30.259238 13289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.131
I20260620 18:42:30.270887 13289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:42967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.131.keytab
--principal=oryx/127.5.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.131:42693
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.225.131
--webserver_port=40319
--tserver_master_addrs=127.5.225.190:34625,127.5.225.189:37095,127.5.225.188:45955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.131
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:30.272280 13289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:30.273653 13289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:30.287530 13295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:30.287563 13294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:30.287603 13297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:30.288199 13289 server_base.cc:1034] running on GCE node
I20260620 18:42:30.288913 13289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:30.289857 13289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:30.291074 13289 hybrid_clock.cc:648] HybridClock initialized: now 1781980950291005 us; error 65 us; skew 500 ppm
I20260620 18:42:30.291415 13289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:30 dist-test-slave-k6lg krb5kdc[12118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980950, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:30.295323 13289 init.cc:377] Logged in from keytab as oryx/127.5.225.131@KRBTEST.COM (short username oryx)
I20260620 18:42:30.296929 13289 webserver.cc:460] Webserver started at http://127.5.225.131:40319/ using document root <none> and password file <none>
I20260620 18:42:30.297770 13289 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:30.297935 13289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:30.303704 13289 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 18:42:30.306557 13304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:30.307964 13289 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:42:30.308159 13289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-2/wal
uuid: "3417abed2a0446919a8150bec84fbafc"
format_stamp: "Formatted at 2026-06-20 18:42:27 on dist-test-slave-k6lg"
server_key: "9629e60360e84005c6a684b42298859b"
server_key_iv: "c991f75e20258c4123f5f96692db1664"
server_key_version: "encryptionkey@0"
I20260620 18:42:30.309247 13289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:30.333133 13289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:30.337399 13289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:30.337719 13289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:30.338824 13289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:30.340137 13289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:30.340265 13289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:30.340391 13289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:30.340474 13289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:30.372766 13289 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.131:42693
I20260620 18:42:30.372852 13416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.131:42693 every 8 connection(s)
I20260620 18:42:30.374282 13289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
I20260620 18:42:30.382017 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 13289
Jun 20 18:42:30 dist-test-slave-k6lg krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980950, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.225.131@KRBTEST.COM for oryx/127.5.225.188@KRBTEST.COM
Jun 20 18:42:30 dist-test-slave-k6lg krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980950, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.225.131@KRBTEST.COM for oryx/127.5.225.189@KRBTEST.COM
I20260620 18:42:30.405134 13421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:30.377163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:42185 (local address 127.5.225.189:37095)
0620 18:42:30.377643 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:30.377651 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:30.379836 (+ 2185us) server_negotiation.cc:407] Connection header received
0620 18:42:30.383368 (+ 3532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:30.383378 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:30.383496 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:30.383607 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:30.386448 (+ 2841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:30.387464 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:30.390033 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:30.390700 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:30.390797 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:30.397904 (+ 7107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:30.397913 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:30.397918 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:30.397962 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:30.401659 (+ 3697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:30.402527 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:30.402792 (+ 265us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:30.402798 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:30.402957 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:30.403427 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:30.403435 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:30.403440 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:30.403775 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:30.403899 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:30.404548 (+ 649us) server_negotiation.cc:299] Negotiation successful
0620 18:42:30.404763 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":392,"threads_started":1}
Jun 20 18:42:30 dist-test-slave-k6lg krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980950, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.225.131@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM
I20260620 18:42:30.406327 13422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:30.377480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:52239 (local address 127.5.225.188:45955)
0620 18:42:30.386655 (+ 9175us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:30.386663 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:30.386687 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:42:30.386786 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:30.386794 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:30.386897 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:30.387016 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:30.388301 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:30.389472 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:30.391468 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:30.392056 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:30.392193 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:30.397107 (+ 4914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:30.397114 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:30.397121 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:30.397163 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:30.401171 (+ 4008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:30.402300 (+ 1129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:30.402308 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:30.402312 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:30.402461 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:30.403001 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:30.403009 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:30.403014 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:30.403388 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:30.403521 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:30.405759 (+ 2238us) server_negotiation.cc:299] Negotiation successful
0620 18:42:30.405970 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9021,"thread_start_us":8911,"threads_started":1}
I20260620 18:42:30.407006 13423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:30.377726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:37095 (local address 127.5.225.131:42185)
0620 18:42:30.379626 (+ 1900us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:30.379686 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:30.382967 (+ 3281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:30.384151 (+ 1184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:30.384167 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:30.385159 (+ 992us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:30.386198 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:30.386230 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:30.387664 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:30.387669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:30.389857 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:30.389869 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:30.391125 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:30.391132 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:30.391474 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:30.392762 (+ 1288us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:30.392794 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:30.397660 (+ 4866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:30.401818 (+ 4158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:30.401823 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:30.401827 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:30.402392 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:30.403107 (+ 715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:30.403116 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:30.403120 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:30.403292 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:30.403977 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:30.403983 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:30.404138 (+ 155us) client_negotiation.cc:769] Sending connection context
0620 18:42:30.405366 (+ 1228us) client_negotiation.cc:240] Negotiation successful
0620 18:42:30.405637 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"mutex_wait_us":21,"spinlock_wait_cycles":7168,"thread_start_us":515,"threads_started":1}
I20260620 18:42:30.407423 13424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:30.377726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:45955 (local address 127.5.225.131:52239)
0620 18:42:30.379625 (+ 1899us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:30.380284 (+ 659us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:30.382967 (+ 2683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:30.387076 (+ 4109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:30.387081 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:30.387757 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:30.388127 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:30.388139 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:30.389661 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:30.389667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:30.391295 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:30.391313 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:30.392257 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:30.392265 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:30.392421 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:30.393047 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:30.393071 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:30.396892 (+ 3821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:30.401361 (+ 4469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:30.401372 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:30.401395 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:30.402121 (+ 726us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:30.402591 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:30.402597 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:30.402602 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:30.402864 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:30.403590 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:30.403599 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:30.404072 (+ 473us) client_negotiation.cc:769] Sending connection context
0620 18:42:30.404419 (+ 347us) client_negotiation.cc:240] Negotiation successful
0620 18:42:30.404963 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1691,"mutex_wait_us":78,"spinlock_wait_cycles":76928,"thread_start_us":1368,"threads_started":1}
I20260620 18:42:30.408978 13418 heartbeater.cc:347] Connected to a master server at 127.5.225.188:45955
I20260620 18:42:30.409039 13419 heartbeater.cc:347] Connected to a master server at 127.5.225.189:37095
I20260620 18:42:30.409327 13419 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:30.409351 13418 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:30.410355 13418 heartbeater.cc:511] Master 127.5.225.188:45955 requested a full tablet report, sending...
I20260620 18:42:30.412652 12950 ts_manager.cc:194] Registered new tserver with Master: 3417abed2a0446919a8150bec84fbafc (127.5.225.131:42693)
I20260620 18:42:30.413421 13419 heartbeater.cc:511] Master 127.5.225.189:37095 requested a full tablet report, sending...
I20260620 18:42:30.413843 13425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:30.379745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:34625 (local address 127.5.225.131:45103)
0620 18:42:30.393368 (+ 13623us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:30.393398 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:30.393632 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:30.394444 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:30.394459 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:30.395438 (+ 979us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:30.395983 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:30.395999 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:30.398014 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:30.398022 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:30.400384 (+ 2362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:30.400404 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:30.401947 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:30.401957 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:30.402269 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:30.402889 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:30.402919 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:30.406607 (+ 3688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:30.410973 (+ 4366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:30.410980 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:30.410984 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:30.411470 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:30.411968 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:30.411973 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:30.411977 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:30.412139 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:30.412819 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:30.412823 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:30.412999 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 18:42:30.413133 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 18:42:30.413446 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13464,"thread_start_us":13309,"threads_started":1}
I20260620 18:42:30.414738 13426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:30.379866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:45103 (local address 127.5.225.190:34625)
0620 18:42:30.392523 (+ 12657us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:30.392531 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:30.393496 (+ 965us) server_negotiation.cc:407] Connection header received
0620 18:42:30.393858 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:30.393868 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:30.393974 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:30.394070 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:30.396210 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:30.397727 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:30.400581 (+ 2854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:30.401486 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:30.401602 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:30.406898 (+ 5296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:30.406907 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:30.406914 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:30.406970 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:30.410812 (+ 3842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:30.411618 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:30.411623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:30.411626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:30.411834 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:30.412286 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:30.412291 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:30.412294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:30.412609 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:30.412757 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:30.414121 (+ 1364us) server_negotiation.cc:299] Negotiation successful
0620 18:42:30.414370 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12488,"thread_start_us":12661,"threads_started":1}
I20260620 18:42:30.414988 12761 ts_manager.cc:194] Registered new tserver with Master: 3417abed2a0446919a8150bec84fbafc (127.5.225.131:42693)
I20260620 18:42:30.416424 13417 heartbeater.cc:347] Connected to a master server at 127.5.225.190:34625
I20260620 18:42:30.416592 13417 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:30.417143 13417 heartbeater.cc:511] Master 127.5.225.190:34625 requested a full tablet report, sending...
I20260620 18:42:30.418081 12163 ts_manager.cc:194] Re-registered known tserver with Master: 3417abed2a0446919a8150bec84fbafc (127.5.225.131:42693)
I20260620 18:42:30.418967 12163 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.225.131@KRBTEST.COM'} at 127.5.225.131:45103
I20260620 18:42:30.438992 13426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:30.410431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59996 (local address 127.5.225.190:34625)
0620 18:42:30.415038 (+ 4607us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:30.415043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:30.422714 (+ 7671us) server_negotiation.cc:407] Connection header received
0620 18:42:30.422885 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:30.422892 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:30.423013 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:30.423115 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:30.425586 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:30.426506 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:30.428329 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:30.428945 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:30.429078 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:30.432813 (+ 3735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:30.432817 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:30.432821 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:30.432853 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:30.435681 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:30.436493 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:30.436500 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:30.436504 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:30.436622 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:30.436957 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:30.436965 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:30.436970 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:30.437343 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:30.437603 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:30.438345 (+ 742us) server_negotiation.cc:299] Negotiation successful
0620 18:42:30.438571 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4500,"thread_start_us":12571,"threads_started":1}
I20260620 18:42:30.440574 13421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:30.411032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47402 (local address 127.5.225.189:37095)
0620 18:42:30.411267 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:30.411275 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:30.423872 (+ 12597us) server_negotiation.cc:407] Connection header received
0620 18:42:30.424169 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:30.424177 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:30.424272 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:30.424353 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:30.427298 (+ 2945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:30.428384 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:30.430009 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:30.430849 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:30.430985 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:30.434357 (+ 3372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:30.434364 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:30.434370 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:30.434414 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:30.437565 (+ 3151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:30.438399 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:30.438404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:30.438407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:30.438571 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:30.438921 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:30.438926 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:30.438929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:30.439290 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:30.439522 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:30.440080 (+ 558us) server_negotiation.cc:299] Negotiation successful
0620 18:42:30.440295 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:42:30.442456 13422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:30.423935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60442 (local address 127.5.225.188:45955)
0620 18:42:30.424116 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:30.424121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:30.425430 (+ 1309us) server_negotiation.cc:407] Connection header received
0620 18:42:30.425647 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:30.425655 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:30.425775 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:30.425871 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:30.427991 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:30.428946 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:30.431125 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:30.431774 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:30.431904 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:30.435274 (+ 3370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:30.435281 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:30.435286 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:30.435328 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:30.438975 (+ 3647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:30.440240 (+ 1265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:30.440245 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:30.440249 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:30.440413 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:30.440918 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:30.440923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:30.440926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:30.441265 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:30.441530 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:30.441909 (+ 379us) server_negotiation.cc:299] Negotiation successful
0620 18:42:30.442131 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:42:30.444128 12163 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59996:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:42:30.450947 12163 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:42:30.478375 13447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:30.468272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:42219 (local address 127.0.0.1:59816)
0620 18:42:30.470245 (+ 1973us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:30.470267 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:30.470445 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:30.471372 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:30.471379 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:30.471405 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:30.471793 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:30.471805 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:30.474213 (+ 2408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:30.474220 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:30.475970 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:30.475982 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:30.477478 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:30.477486 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:30.477676 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:30.477710 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 18:42:30.477798 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 18:42:30.477945 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1808,"mutex_wait_us":51,"thread_start_us":2389,"threads_started":1}
I20260620 18:42:30.478768 13448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:30.468932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56128 (local address 127.5.225.130:45529)
0620 18:42:30.469437 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:30.469443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:30.471054 (+ 1611us) server_negotiation.cc:407] Connection header received
0620 18:42:30.471231 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:30.471242 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:30.471477 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:30.471682 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:30.472294 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:30.473561 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:30.476374 (+ 2813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:30.477525 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:30.477628 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:30.477838 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:30.478300 (+ 462us) server_negotiation.cc:299] Negotiation successful
0620 18:42:30.478470 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":241,"threads_started":1}
I20260620 18:42:30.479131 13446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:30.468743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59816 (local address 127.5.225.129:42219)
0620 18:42:30.469837 (+ 1094us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:30.469843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:30.470349 (+ 506us) server_negotiation.cc:407] Connection header received
0620 18:42:30.470618 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:30.470629 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:30.470928 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:30.471127 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:30.471988 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:30.474028 (+ 2040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:30.476109 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:30.477316 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:30.477453 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:30.477677 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:30.478687 (+ 1010us) server_negotiation.cc:299] Negotiation successful
0620 18:42:30.478881 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":929,"thread_start_us":825,"threads_started":1}
I20260620 18:42:30.479569 13445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:30.468721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.130:45529 (local address 127.0.0.1:56128)
0620 18:42:30.470901 (+ 2180us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:30.470917 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:30.471055 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:30.471666 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:30.471670 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:30.471709 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:30.472095 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:30.472106 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:30.473755 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:30.473762 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:30.476193 (+ 2431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:30.476213 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:30.477877 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:30.477882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:30.478053 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:30.478088 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 18:42:30.479017 (+ 929us) client_negotiation.cc:240] Negotiation successful
0620 18:42:30.479157 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2051,"mutex_wait_us":2,"thread_start_us":1382,"threads_started":1}
I20260620 18:42:30.480019 13443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:30.467909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.131:42693 (local address 127.0.0.1:45628)
0620 18:42:30.468437 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:30.468458 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:30.468577 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:30.472122 (+ 3545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:30.472133 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:30.472159 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:30.472540 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:30.472552 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:30.474409 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:30.474415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:30.476590 (+ 2175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:30.476602 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:30.479069 (+ 2467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:30.479075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:30.479244 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:30.479289 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 18:42:30.479353 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 18:42:30.479475 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":757,"threads_started":1}
I20260620 18:42:30.480434 13444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:30.468178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45628 (local address 127.5.225.131:42693)
0620 18:42:30.471590 (+ 3412us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:30.471599 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:30.471634 (+ 35us) server_negotiation.cc:407] Connection header received
0620 18:42:30.471726 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:30.471734 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:30.471947 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:30.472126 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:30.472745 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:30.474279 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:30.476749 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:30.478838 (+ 2089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:30.479680 (+ 842us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:30.479908 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:30.479977 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 18:42:30.480126 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3247,"thread_start_us":3082,"threads_started":1}
I20260620 18:42:30.482951 13211 tablet_service.cc:1467] Processing CreateTablet for tablet 8e7a18e9cbb04e2e9d0690ae3bcefcc3 (DEFAULT_TABLE table=test-table [id=628e1f9654bc467aa640c341b8c79c67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:42:30.483011 13063 tablet_service.cc:1467] Processing CreateTablet for tablet 8e7a18e9cbb04e2e9d0690ae3bcefcc3 (DEFAULT_TABLE table=test-table [id=628e1f9654bc467aa640c341b8c79c67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:42:30.484198 13211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e7a18e9cbb04e2e9d0690ae3bcefcc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:30.484350 13063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e7a18e9cbb04e2e9d0690ae3bcefcc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:30.483889 13352 tablet_service.cc:1467] Processing CreateTablet for tablet 8e7a18e9cbb04e2e9d0690ae3bcefcc3 (DEFAULT_TABLE table=test-table [id=628e1f9654bc467aa640c341b8c79c67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:42:30.486065 13352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e7a18e9cbb04e2e9d0690ae3bcefcc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:30.496532 13449 tablet_bootstrap.cc:492] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 762e84cc86204351bc255b1e796e833f: Bootstrap starting.
I20260620 18:42:30.499236 13450 tablet_bootstrap.cc:492] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 911d8be242834719ac569e965601a2e2: Bootstrap starting.
I20260620 18:42:30.500097 13449 tablet_bootstrap.cc:654] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 762e84cc86204351bc255b1e796e833f: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:30.501454 13449 log.cc:824] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 762e84cc86204351bc255b1e796e833f: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:30.502699 13450 tablet_bootstrap.cc:654] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 911d8be242834719ac569e965601a2e2: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:30.503916 13450 log.cc:824] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 911d8be242834719ac569e965601a2e2: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:30.504752 13449 tablet_bootstrap.cc:492] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 762e84cc86204351bc255b1e796e833f: No bootstrap required, opened a new log
I20260620 18:42:30.505016 13449 ts_tablet_manager.cc:1397] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 762e84cc86204351bc255b1e796e833f: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260620 18:42:30.505014 13452 tablet_bootstrap.cc:492] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 3417abed2a0446919a8150bec84fbafc: Bootstrap starting.
I20260620 18:42:30.506482 13450 tablet_bootstrap.cc:492] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 911d8be242834719ac569e965601a2e2: No bootstrap required, opened a new log
I20260620 18:42:30.506737 13450 ts_tablet_manager.cc:1397] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 911d8be242834719ac569e965601a2e2: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 18:42:30.508319 13452 tablet_bootstrap.cc:654] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 3417abed2a0446919a8150bec84fbafc: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:30.509608 13452 log.cc:824] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 3417abed2a0446919a8150bec84fbafc: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:30.514869 13452 tablet_bootstrap.cc:492] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 3417abed2a0446919a8150bec84fbafc: No bootstrap required, opened a new log
I20260620 18:42:30.514703 13450 raft_consensus.cc:348] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 911d8be242834719ac569e965601a2e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3417abed2a0446919a8150bec84fbafc" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 42693 } } peers { permanent_uuid: "911d8be242834719ac569e965601a2e2" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 42219 } } peers { permanent_uuid: "762e84cc86204351bc255b1e796e833f" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 45529 } }
I20260620 18:42:30.515060 13450 raft_consensus.cc:374] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 911d8be242834719ac569e965601a2e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:30.515125 13452 ts_tablet_manager.cc:1397] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 3417abed2a0446919a8150bec84fbafc: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20260620 18:42:30.515156 13450 raft_consensus.cc:729] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 911d8be242834719ac569e965601a2e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 911d8be242834719ac569e965601a2e2, State: Initialized, Role: FOLLOWER
I20260620 18:42:30.515856 13450 consensus_queue.cc:260] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 911d8be242834719ac569e965601a2e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3417abed2a0446919a8150bec84fbafc" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 42693 } } peers { permanent_uuid: "911d8be242834719ac569e965601a2e2" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 42219 } } peers { permanent_uuid: "762e84cc86204351bc255b1e796e833f" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 45529 } }
I20260620 18:42:30.516047 13449 raft_consensus.cc:348] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 762e84cc86204351bc255b1e796e833f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3417abed2a0446919a8150bec84fbafc" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 42693 } } peers { permanent_uuid: "911d8be242834719ac569e965601a2e2" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 42219 } } peers { permanent_uuid: "762e84cc86204351bc255b1e796e833f" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 45529 } }
I20260620 18:42:30.516455 13449 raft_consensus.cc:374] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 762e84cc86204351bc255b1e796e833f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:30.516557 13449 raft_consensus.cc:729] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 762e84cc86204351bc255b1e796e833f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 762e84cc86204351bc255b1e796e833f, State: Initialized, Role: FOLLOWER
I20260620 18:42:30.517117 13449 consensus_queue.cc:260] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 762e84cc86204351bc255b1e796e833f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3417abed2a0446919a8150bec84fbafc" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 42693 } } peers { permanent_uuid: "911d8be242834719ac569e965601a2e2" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 42219 } } peers { permanent_uuid: "762e84cc86204351bc255b1e796e833f" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 45529 } }
I20260620 18:42:30.517608 13450 ts_tablet_manager.cc:1428] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 911d8be242834719ac569e965601a2e2: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260620 18:42:30.526899 13276 heartbeater.cc:503] Master 127.5.225.190:34625 was elected leader, sending a full tablet report...
I20260620 18:42:30.528383 13452 raft_consensus.cc:348] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 3417abed2a0446919a8150bec84fbafc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3417abed2a0446919a8150bec84fbafc" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 42693 } } peers { permanent_uuid: "911d8be242834719ac569e965601a2e2" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 42219 } } peers { permanent_uuid: "762e84cc86204351bc255b1e796e833f" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 45529 } }
I20260620 18:42:30.528852 13452 raft_consensus.cc:374] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 3417abed2a0446919a8150bec84fbafc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:30.528923 13449 ts_tablet_manager.cc:1428] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 762e84cc86204351bc255b1e796e833f: Time spent starting tablet: real 0.024s user 0.017s sys 0.002s
I20260620 18:42:30.528962 13452 raft_consensus.cc:729] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 3417abed2a0446919a8150bec84fbafc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3417abed2a0446919a8150bec84fbafc, State: Initialized, Role: FOLLOWER
I20260620 18:42:30.529628 13452 consensus_queue.cc:260] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 3417abed2a0446919a8150bec84fbafc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3417abed2a0446919a8150bec84fbafc" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 42693 } } peers { permanent_uuid: "911d8be242834719ac569e965601a2e2" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 42219 } } peers { permanent_uuid: "762e84cc86204351bc255b1e796e833f" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 45529 } }
I20260620 18:42:30.530790 13417 heartbeater.cc:503] Master 127.5.225.190:34625 was elected leader, sending a full tablet report...
I20260620 18:42:30.531538 13452 ts_tablet_manager.cc:1428] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 3417abed2a0446919a8150bec84fbafc: Time spent starting tablet: real 0.016s user 0.014s sys 0.001s
W20260620 18:42:30.546586 13285 tablet.cc:2369] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 762e84cc86204351bc255b1e796e833f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:42:30.627732 13420 tablet.cc:2369] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 3417abed2a0446919a8150bec84fbafc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:42:30.667212 13131 tablet.cc:2369] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 911d8be242834719ac569e965601a2e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:42:30.809981 13456 raft_consensus.cc:482] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 762e84cc86204351bc255b1e796e833f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:42:30.810308 13456 raft_consensus.cc:504] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 762e84cc86204351bc255b1e796e833f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3417abed2a0446919a8150bec84fbafc" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 42693 } } peers { permanent_uuid: "911d8be242834719ac569e965601a2e2" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 42219 } } peers { permanent_uuid: "762e84cc86204351bc255b1e796e833f" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 45529 } }
I20260620 18:42:30.811915 13456 leader_election.cc:288] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 762e84cc86204351bc255b1e796e833f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3417abed2a0446919a8150bec84fbafc (127.5.225.131:42693), 911d8be242834719ac569e965601a2e2 (127.5.225.129:42219)
I20260620 18:42:30.818622 13444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:30.812185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:33177 (local address 127.5.225.131:42693)
0620 18:42:30.812386 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:30.812394 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:30.813097 (+ 703us) server_negotiation.cc:407] Connection header received
0620 18:42:30.813321 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:30.813329 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:30.813402 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:30.813489 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:30.814181 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:30.814992 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:30.816886 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:30.817637 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:30.817812 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:30.817897 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:30.818240 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 18:42:30.818365 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:42:30.818634 13446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:30.812351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:53881 (local address 127.5.225.129:42219)
0620 18:42:30.812547 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:30.812552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:30.813366 (+ 814us) server_negotiation.cc:407] Connection header received
0620 18:42:30.813546 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:30.813551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:30.813623 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:30.813711 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:30.814320 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:30.814997 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:30.816885 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:30.817659 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:30.817828 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:30.817912 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:30.818257 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 18:42:30.818377 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 18:42:30.818828 13458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:30.812167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:42219 (local address 127.5.225.130:53881)
0620 18:42:30.813258 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:30.813272 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:30.813425 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:30.813737 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:30.813741 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:30.813764 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:30.814130 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:30.814139 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:30.815116 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:30.815121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:30.816743 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:30.816754 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:30.817903 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:30.817909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:30.818078 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:30.818099 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 18:42:30.818214 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 18:42:30.818354 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":992,"mutex_wait_us":51,"thread_start_us":427,"threads_started":1}
I20260620 18:42:30.818851 13459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:30.812108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.131:42693 (local address 127.5.225.130:33177)
0620 18:42:30.812985 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:30.813007 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:30.813169 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:30.813517 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:30.813522 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:30.813548 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:30.813962 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:30.813976 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:30.815115 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:30.815122 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:30.816741 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:30.816753 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:30.817871 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:30.817881 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:30.818029 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:30.818083 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 18:42:30.818205 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 18:42:30.818363 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"mutex_wait_us":96,"thread_start_us":915,"threads_started":1}
I20260620 18:42:30.819747 13083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e7a18e9cbb04e2e9d0690ae3bcefcc3" candidate_uuid: "762e84cc86204351bc255b1e796e833f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "911d8be242834719ac569e965601a2e2" is_pre_election: true
I20260620 18:42:30.819869 13372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e7a18e9cbb04e2e9d0690ae3bcefcc3" candidate_uuid: "762e84cc86204351bc255b1e796e833f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3417abed2a0446919a8150bec84fbafc" is_pre_election: true
I20260620 18:42:30.820261 13083 raft_consensus.cc:2454] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 911d8be242834719ac569e965601a2e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 762e84cc86204351bc255b1e796e833f in term 0.
I20260620 18:42:30.820314 13372 raft_consensus.cc:2454] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 3417abed2a0446919a8150bec84fbafc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 762e84cc86204351bc255b1e796e833f in term 0.
I20260620 18:42:30.820875 13158 leader_election.cc:302] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 762e84cc86204351bc255b1e796e833f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3417abed2a0446919a8150bec84fbafc, 762e84cc86204351bc255b1e796e833f; no voters:
I20260620 18:42:30.821282 13456 raft_consensus.cc:2781] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 762e84cc86204351bc255b1e796e833f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:42:30.821415 13456 raft_consensus.cc:482] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 762e84cc86204351bc255b1e796e833f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:42:30.821522 13456 raft_consensus.cc:3037] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 762e84cc86204351bc255b1e796e833f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:30.823701 13456 raft_consensus.cc:504] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 762e84cc86204351bc255b1e796e833f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3417abed2a0446919a8150bec84fbafc" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 42693 } } peers { permanent_uuid: "911d8be242834719ac569e965601a2e2" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 42219 } } peers { permanent_uuid: "762e84cc86204351bc255b1e796e833f" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 45529 } }
I20260620 18:42:30.824280 13456 leader_election.cc:288] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 762e84cc86204351bc255b1e796e833f [CANDIDATE]: Term 1 election: Requested vote from peers 3417abed2a0446919a8150bec84fbafc (127.5.225.131:42693), 911d8be242834719ac569e965601a2e2 (127.5.225.129:42219)
I20260620 18:42:30.824664 13372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e7a18e9cbb04e2e9d0690ae3bcefcc3" candidate_uuid: "762e84cc86204351bc255b1e796e833f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3417abed2a0446919a8150bec84fbafc"
I20260620 18:42:30.824704 13083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e7a18e9cbb04e2e9d0690ae3bcefcc3" candidate_uuid: "762e84cc86204351bc255b1e796e833f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "911d8be242834719ac569e965601a2e2"
I20260620 18:42:30.824855 13372 raft_consensus.cc:3037] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 3417abed2a0446919a8150bec84fbafc [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:30.824883 13083 raft_consensus.cc:3037] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 911d8be242834719ac569e965601a2e2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:30.827003 13372 raft_consensus.cc:2454] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 3417abed2a0446919a8150bec84fbafc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 762e84cc86204351bc255b1e796e833f in term 1.
I20260620 18:42:30.827051 13083 raft_consensus.cc:2454] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 911d8be242834719ac569e965601a2e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 762e84cc86204351bc255b1e796e833f in term 1.
I20260620 18:42:30.827385 13158 leader_election.cc:302] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 762e84cc86204351bc255b1e796e833f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3417abed2a0446919a8150bec84fbafc, 762e84cc86204351bc255b1e796e833f; no voters:
I20260620 18:42:30.827663 13456 raft_consensus.cc:2781] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 762e84cc86204351bc255b1e796e833f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:42:30.828095 13456 raft_consensus.cc:686] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 762e84cc86204351bc255b1e796e833f [term 1 LEADER]: Becoming Leader. State: Replica: 762e84cc86204351bc255b1e796e833f, State: Running, Role: LEADER
I20260620 18:42:30.828624 13456 consensus_queue.cc:237] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 762e84cc86204351bc255b1e796e833f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3417abed2a0446919a8150bec84fbafc" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 42693 } } peers { permanent_uuid: "911d8be242834719ac569e965601a2e2" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 42219 } } peers { permanent_uuid: "762e84cc86204351bc255b1e796e833f" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 45529 } }
I20260620 18:42:30.833739 12162 catalog_manager.cc:5495] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 762e84cc86204351bc255b1e796e833f reported cstate change: term changed from 0 to 1, leader changed from <none> to 762e84cc86204351bc255b1e796e833f (127.5.225.130). New cstate: current_term: 1 leader_uuid: "762e84cc86204351bc255b1e796e833f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3417abed2a0446919a8150bec84fbafc" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 42693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "911d8be242834719ac569e965601a2e2" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 42219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "762e84cc86204351bc255b1e796e833f" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 45529 } health_report { overall_health: HEALTHY } } }
I20260620 18:42:30.935076 13448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:30.930139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56136 (local address 127.5.225.130:45529)
0620 18:42:30.930373 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:30.930378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:30.930395 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:42:30.930560 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:30.930565 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:30.930649 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:30.930773 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:42:30.931094 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:30.931783 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:30.933093 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:30.933499 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:30.933591 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:30.933963 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:42:30.934530 (+ 567us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:42:30.934637 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:30.934715 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 18:42:30.934776 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:42:30.943317 13372 raft_consensus.cc:1261] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 3417abed2a0446919a8150bec84fbafc [term 1 FOLLOWER]: Refusing update from remote peer 762e84cc86204351bc255b1e796e833f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:42:30.943317 13083 raft_consensus.cc:1261] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 911d8be242834719ac569e965601a2e2 [term 1 FOLLOWER]: Refusing update from remote peer 762e84cc86204351bc255b1e796e833f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:42:30.944428 13456 consensus_queue.cc:1035] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 762e84cc86204351bc255b1e796e833f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3417abed2a0446919a8150bec84fbafc" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 42693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:42:30.944794 13460 consensus_queue.cc:1035] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 762e84cc86204351bc255b1e796e833f [LEADER]: Connected to new peer: Peer: permanent_uuid: "911d8be242834719ac569e965601a2e2" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 42219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:42:30.963820 13464 mvcc.cc:205] Tried to move back new op lower bound from 7298993975053762560 to 7298993974597632000. Current Snapshot: MvccSnapshot[applied={T|T < 7298993975053762560}]
I20260620 18:42:30.976888 12162 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:42:30.977146 12162 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:42:30.985399 12162 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6cf69d27a27142a9b0c07415bb23d249: Sending DeleteTablet for 3 replicas of tablet 8e7a18e9cbb04e2e9d0690ae3bcefcc3
I20260620 18:42:30.986754 13211 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e7a18e9cbb04e2e9d0690ae3bcefcc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:42:30 UTC) from {username='oryx', principal='oryx/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:56128
I20260620 18:42:30.987180 13063 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e7a18e9cbb04e2e9d0690ae3bcefcc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:42:30 UTC) from {username='oryx', principal='oryx/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:59816
I20260620 18:42:30.998864 13474 tablet_replica.cc:330] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 911d8be242834719ac569e965601a2e2: stopping tablet replica
I20260620 18:42:30.999415 13474 raft_consensus.cc:2229] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 911d8be242834719ac569e965601a2e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:42:30.999994 13474 raft_consensus.cc:2258] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 911d8be242834719ac569e965601a2e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:42:31.001999 13474 ts_tablet_manager.cc:1905] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 911d8be242834719ac569e965601a2e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:42:31.009428 13474 ts_tablet_manager.cc:1918] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 911d8be242834719ac569e965601a2e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:42:31.009681 13474 log.cc:1196] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 911d8be242834719ac569e965601a2e2: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-0/wal/wals/8e7a18e9cbb04e2e9d0690ae3bcefcc3
I20260620 18:42:31.009847 13473 tablet_replica.cc:330] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 762e84cc86204351bc255b1e796e833f: stopping tablet replica
I20260620 18:42:31.010272 13473 raft_consensus.cc:2229] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 762e84cc86204351bc255b1e796e833f [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:42:31.010491 13474 ts_tablet_manager.cc:1939] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 911d8be242834719ac569e965601a2e2: Deleting consensus metadata
I20260620 18:42:31.010928 13473 raft_consensus.cc:2258] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 762e84cc86204351bc255b1e796e833f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:42:31.012410 12148 catalog_manager.cc:4841] TS 911d8be242834719ac569e965601a2e2 (127.5.225.129:42219): tablet 8e7a18e9cbb04e2e9d0690ae3bcefcc3 (table test-table [id=628e1f9654bc467aa640c341b8c79c67]) successfully deleted
I20260620 18:42:31.012929 13473 ts_tablet_manager.cc:1905] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 762e84cc86204351bc255b1e796e833f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:42:31.021802 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 12131
I20260620 18:42:31.023059 12193 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:31.030117 13473 ts_tablet_manager.cc:1918] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 762e84cc86204351bc255b1e796e833f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:42:31.030370 13473 log.cc:1196] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 762e84cc86204351bc255b1e796e833f: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-1/wal/wals/8e7a18e9cbb04e2e9d0690ae3bcefcc3
I20260620 18:42:31.030903 13473 ts_tablet_manager.cc:1939] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 762e84cc86204351bc255b1e796e833f: Deleting consensus metadata
I20260620 18:42:31.041297 13476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:30.990053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47408 (local address 127.5.225.189:37095)
0620 18:42:31.005461 (+ 15408us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:31.005468 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:31.013449 (+ 7981us) server_negotiation.cc:407] Connection header received
0620 18:42:31.013701 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:31.013706 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:31.013790 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:31.013899 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:31.016137 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:31.017412 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:31.019668 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:31.020329 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:31.020430 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:31.034375 (+ 13945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:31.034383 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:31.034389 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:31.034436 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:31.038176 (+ 3740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:31.039112 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:31.039118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:31.039122 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:31.039286 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:31.039710 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:31.039715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:31.039719 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:31.040090 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:31.040351 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:31.040675 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 18:42:31.040904 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15271,"thread_start_us":15161,"threads_started":1}
I20260620 18:42:31.057808 13477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:31.014406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60454 (local address 127.5.225.188:45955)
0620 18:42:31.025724 (+ 11318us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:31.025730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:31.041397 (+ 15667us) server_negotiation.cc:407] Connection header received
0620 18:42:31.041639 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:31.041644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:31.041722 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:31.041798 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:31.043646 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:31.044859 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:31.046789 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:31.047442 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:31.047555 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:31.051248 (+ 3693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:31.051255 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:31.051260 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:31.051300 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:31.054822 (+ 3522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:31.055603 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:31.055609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:31.055613 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:31.055763 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:31.056197 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:31.056202 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:31.056206 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:31.056554 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:31.056785 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:31.057181 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 18:42:31.057398 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11156,"thread_start_us":11315,"threads_started":1}
I20260620 18:42:31.154433 12147 catalog_manager.cc:4841] TS 762e84cc86204351bc255b1e796e833f (127.5.225.130:45529): tablet 8e7a18e9cbb04e2e9d0690ae3bcefcc3 (table test-table [id=628e1f9654bc467aa640c341b8c79c67]) successfully deleted
I20260620 18:42:31.155012 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 12131
I20260620 18:42:31.205682 13476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:31.184984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47424 (local address 127.5.225.189:37095)
0620 18:42:31.185306 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:31.185313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:31.189523 (+ 4210us) server_negotiation.cc:407] Connection header received
0620 18:42:31.189758 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:31.189767 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:31.189863 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:31.189984 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:31.192258 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:31.193371 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:31.194852 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:31.195525 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:31.195660 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:31.199062 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:31.199067 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:31.199070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:31.199103 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:31.202972 (+ 3869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:31.203725 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:31.203731 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:31.203735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:31.203867 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:31.204286 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:31.204291 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:31.204294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:31.204547 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:31.204814 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:31.205195 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 18:42:31.205408 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 18:42:31.207191 13477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:31.190336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60462 (local address 127.5.225.188:45955)
0620 18:42:31.190594 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:31.190598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:31.190931 (+ 333us) server_negotiation.cc:407] Connection header received
0620 18:42:31.191159 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:31.191164 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:31.191283 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:31.191389 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:31.193257 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:31.194337 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:31.196613 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:31.197296 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:31.197449 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:31.200596 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:31.200604 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:31.200610 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:31.200657 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:31.203832 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:31.204704 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:31.204709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:31.204712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:31.204841 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:31.205218 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:31.205223 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:31.205227 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:31.205596 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:31.206088 (+ 492us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:31.206658 (+ 570us) server_negotiation.cc:299] Negotiation successful
0620 18:42:31.206865 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 18:42:31.965150 13485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:31.964325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:34625 (local address 127.5.225.130:38923)
0620 18:42:31.964769 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:31.964936 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:34625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":228,"threads_started":1}
W20260620 18:42:31.965675 13276 heartbeater.cc:644] Failed to heartbeat to 127.5.225.190:34625 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.225.190:34625: connect: Connection refused (error 111)
I20260620 18:42:31.968796 13486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:31.968170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:34625 (local address 127.5.225.131:33429)
0620 18:42:31.968542 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:31.968644 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:34625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":195,"threads_started":1}
W20260620 18:42:31.969208 13417 heartbeater.cc:644] Failed to heartbeat to 127.5.225.190:34625 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.225.190:34625: connect: Connection refused (error 111)
I20260620 18:42:31.971691 13487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:31.970903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:34625 (local address 127.5.225.129:37965)
0620 18:42:31.971426 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:31.971552 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:34625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":305,"threads_started":1}
W20260620 18:42:31.972041 13128 heartbeater.cc:644] Failed to heartbeat to 127.5.225.190:34625 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.225.190:34625: connect: Connection refused (error 111)
I20260620 18:42:32.460703 13488 raft_consensus.cc:482] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 3417abed2a0446919a8150bec84fbafc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 762e84cc86204351bc255b1e796e833f)
I20260620 18:42:32.460937 13488 raft_consensus.cc:504] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 3417abed2a0446919a8150bec84fbafc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3417abed2a0446919a8150bec84fbafc" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 42693 } } peers { permanent_uuid: "911d8be242834719ac569e965601a2e2" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 42219 } } peers { permanent_uuid: "762e84cc86204351bc255b1e796e833f" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 45529 } }
I20260620 18:42:32.462391 13488 leader_election.cc:288] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 3417abed2a0446919a8150bec84fbafc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 911d8be242834719ac569e965601a2e2 (127.5.225.129:42219), 762e84cc86204351bc255b1e796e833f (127.5.225.130:45529)
I20260620 18:42:32.471035 13491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:32.462873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:49639 (local address 127.5.225.130:45529)
0620 18:42:32.464221 (+ 1348us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:32.464227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:32.464246 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:42:32.464313 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:32.464318 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:32.464388 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:32.464522 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:32.465059 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:32.466037 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:32.468431 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:32.469646 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:32.469827 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:32.469905 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:32.470573 (+ 668us) server_negotiation.cc:299] Negotiation successful
0620 18:42:32.470717 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1211,"thread_start_us":1107,"threads_started":1}
I20260620 18:42:32.471031 13490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:32.462731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:51001 (local address 127.5.225.129:42219)
0620 18:42:32.463790 (+ 1059us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:32.463798 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:32.463820 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:42:32.463897 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:32.463902 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:32.464001 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:32.464153 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:32.464679 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:32.465732 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:32.468448 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:32.469711 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:32.469857 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:32.469919 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:32.470573 (+ 654us) server_negotiation.cc:299] Negotiation successful
0620 18:42:32.470729 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":849,"thread_start_us":733,"threads_started":1}
I20260620 18:42:32.471066 13486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:32.462642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:42219 (local address 127.5.225.131:51001)
0620 18:42:32.462957 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:32.462986 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:32.463156 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:32.464198 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:32.464203 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:32.464236 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:32.464487 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:32.464501 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:32.465903 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:32.465907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:32.468270 (+ 2363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:32.468285 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:32.469890 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:32.469896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:32.470262 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:32.470415 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 18:42:32.470510 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 18:42:32.470717 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":197}
I20260620 18:42:32.471138 13489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:32.462687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.130:45529 (local address 127.5.225.131:49639)
0620 18:42:32.463899 (+ 1212us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:32.463916 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:32.464037 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:32.464562 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:32.464567 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:32.464593 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:32.464877 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:32.464891 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:32.466217 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:32.466223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:32.468270 (+ 2047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:32.468285 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:32.469890 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:32.469896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:32.470260 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:32.470417 (+ 157us) client_negotiation.cc:769] Sending connection context
0620 18:42:32.470509 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 18:42:32.470717 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1093,"mutex_wait_us":241,"thread_start_us":990,"threads_started":1}
W20260620 18:42:32.472718 13306 leader_election.cc:341] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 3417abed2a0446919a8150bec84fbafc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 911d8be242834719ac569e965601a2e2 (127.5.225.129:42219): Not found: Tablet not found: 8e7a18e9cbb04e2e9d0690ae3bcefcc3
W20260620 18:42:32.473042 13305 leader_election.cc:341] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 3417abed2a0446919a8150bec84fbafc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 762e84cc86204351bc255b1e796e833f (127.5.225.130:45529): Not found: Tablet not found: 8e7a18e9cbb04e2e9d0690ae3bcefcc3
I20260620 18:42:32.473201 13305 leader_election.cc:302] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 3417abed2a0446919a8150bec84fbafc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3417abed2a0446919a8150bec84fbafc; no voters: 762e84cc86204351bc255b1e796e833f, 911d8be242834719ac569e965601a2e2
I20260620 18:42:32.473573 13488 raft_consensus.cc:2726] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 3417abed2a0446919a8150bec84fbafc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 18:42:32.481839 13492 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6cf69d27a27142a9b0c07415bb23d249)
I20260620 18:42:32.482227 13492 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: false } } peers { permanent_uuid: "c2efcc70af1e4061b9d972d5b56de43d" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 45955 } attrs { promote: false } }
I20260620 18:42:32.483754 13492 leader_election.cc:288] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6cf69d27a27142a9b0c07415bb23d249 (127.5.225.190:34625), 45a9ce1454f24d488dd7295235c75786 (127.5.225.189:37095)
I20260620 18:42:32.485188 13494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:32.483918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:34625 (local address 127.0.0.1:60412)
0620 18:42:32.484558 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:32.484855 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:34625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":269,"threads_started":1}
W20260620 18:42:32.485446 12936 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.225.190:34625: connect: Connection refused (error 111)
I20260620 18:42:32.486635 13497 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6cf69d27a27142a9b0c07415bb23d249)
I20260620 18:42:32.487140 13494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:32.486816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:34625 (local address 127.0.0.1:60426)
0620 18:42:32.486954 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:32.487027 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:34625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 18:42:32.486907 13497 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: false } } peers { permanent_uuid: "c2efcc70af1e4061b9d972d5b56de43d" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 45955 } attrs { promote: false } }
W20260620 18:42:32.487380 12936 leader_election.cc:334] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6cf69d27a27142a9b0c07415bb23d249 (127.5.225.190:34625): Network error: Client connection negotiation failed: client connection to 127.5.225.190:34625: connect: Connection refused (error 111)
I20260620 18:42:32.488430 13497 leader_election.cc:288] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6cf69d27a27142a9b0c07415bb23d249 (127.5.225.190:34625), c2efcc70af1e4061b9d972d5b56de43d (127.5.225.188:45955)
I20260620 18:42:32.489426 13498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:32.488572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:34625 (local address 127.0.0.1:60440)
0620 18:42:32.488937 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:32.489143 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:34625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":189,"threads_started":1}
W20260620 18:42:32.491345 12747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.225.190:34625: connect: Connection refused (error 111)
I20260620 18:42:32.492601 13493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:32.483918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:37095 (local address 127.0.0.1:47432)
0620 18:42:32.485053 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:32.485069 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:32.485236 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:32.485598 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:32.485603 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:32.485630 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:32.486060 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:32.486075 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:32.487626 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:32.487630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:32.489643 (+ 2013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:32.489661 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:32.491781 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:32.491787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:32.491954 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:32.491998 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 18:42:32.492101 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 18:42:32.492250 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1012,"mutex_wait_us":45,"thread_start_us":796,"threads_started":1}
I20260620 18:42:32.493290 13495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:32.484057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47432 (local address 127.5.225.189:37095)
0620 18:42:32.484850 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:32.484856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:32.485162 (+ 306us) server_negotiation.cc:407] Connection header received
0620 18:42:32.485391 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:32.485395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:32.485455 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:32.485537 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:32.486330 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:32.487501 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:32.489893 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:32.491599 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:32.491769 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:32.491921 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:32.492808 (+ 887us) server_negotiation.cc:299] Negotiation successful
0620 18:42:32.492946 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":429,"threads_started":1}
I20260620 18:42:32.493887 12770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2efcc70af1e4061b9d972d5b56de43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "45a9ce1454f24d488dd7295235c75786" is_pre_election: true
I20260620 18:42:32.494243 12770 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2efcc70af1e4061b9d972d5b56de43d in term 1.
I20260620 18:42:32.494771 12935 leader_election.cc:302] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 45a9ce1454f24d488dd7295235c75786, c2efcc70af1e4061b9d972d5b56de43d; no voters: 6cf69d27a27142a9b0c07415bb23d249
I20260620 18:42:32.495187 13492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 18:42:32.495368 13492 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6cf69d27a27142a9b0c07415bb23d249)
I20260620 18:42:32.495494 13492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:42:32.495668 13499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:32.493100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:34625 (local address 127.0.0.1:60448)
0620 18:42:32.493309 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:32.493634 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:34625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 18:42:32.495836 12747 leader_election.cc:334] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6cf69d27a27142a9b0c07415bb23d249 (127.5.225.190:34625): Network error: Client connection negotiation failed: client connection to 127.5.225.190:34625: connect: Connection refused (error 111)
I20260620 18:42:32.496905 13498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:32.489101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:45955 (local address 127.0.0.1:60478)
0620 18:42:32.489704 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:32.489724 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:32.489855 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:32.490238 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:32.490243 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:32.490270 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:32.490715 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:32.490726 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:32.492189 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:32.492194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:32.494375 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:32.494389 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:32.496114 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:32.496120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:32.496304 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:32.496328 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 18:42:32.496435 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 18:42:32.496590 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":1936,"threads_started":1}
I20260620 18:42:32.497188 13500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:32.489315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60478 (local address 127.5.225.188:45955)
0620 18:42:32.489954 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:32.489959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:32.489978 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:42:32.490032 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:32.490041 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:32.490094 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:32.490229 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:32.490861 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:32.492045 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:32.494549 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:32.495927 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:32.496080 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:32.496203 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:32.496458 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 18:42:32.496600 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":384,"threads_started":1}
I20260620 18:42:32.497720 12960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45a9ce1454f24d488dd7295235c75786" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c2efcc70af1e4061b9d972d5b56de43d" is_pre_election: true
I20260620 18:42:32.498656 13492 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: false } } peers { permanent_uuid: "c2efcc70af1e4061b9d972d5b56de43d" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 45955 } attrs { promote: false } }
I20260620 18:42:32.499073 12960 raft_consensus.cc:2379] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 45a9ce1454f24d488dd7295235c75786 in current term 2: Already voted for candidate c2efcc70af1e4061b9d972d5b56de43d in this term.
I20260620 18:42:32.499306 13492 leader_election.cc:288] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [CANDIDATE]: Term 2 election: Requested vote from peers 6cf69d27a27142a9b0c07415bb23d249 (127.5.225.190:34625), 45a9ce1454f24d488dd7295235c75786 (127.5.225.189:37095)
I20260620 18:42:32.499754 12770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2efcc70af1e4061b9d972d5b56de43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "45a9ce1454f24d488dd7295235c75786"
I20260620 18:42:32.499935 13493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:32.499435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:34625 (local address 127.0.0.1:60454)
0620 18:42:32.499685 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:32.499777 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:34625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260620 18:42:32.499975 12770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:42:32.500563 12747 leader_election.cc:302] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45a9ce1454f24d488dd7295235c75786; no voters: 6cf69d27a27142a9b0c07415bb23d249, c2efcc70af1e4061b9d972d5b56de43d
I20260620 18:42:32.500948 13493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:32.500554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:34625 (local address 127.0.0.1:60462)
0620 18:42:32.500706 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:32.500798 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:34625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 18:42:32.501134 12936 leader_election.cc:334] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6cf69d27a27142a9b0c07415bb23d249 (127.5.225.190:34625): Network error: Client connection negotiation failed: client connection to 127.5.225.190:34625: connect: Connection refused (error 111)
I20260620 18:42:32.502530 12770 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2efcc70af1e4061b9d972d5b56de43d in term 2.
I20260620 18:42:32.502725 13497 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 18:42:32.502936 12935 leader_election.cc:302] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 45a9ce1454f24d488dd7295235c75786, c2efcc70af1e4061b9d972d5b56de43d; no voters: 6cf69d27a27142a9b0c07415bb23d249
I20260620 18:42:32.503279 13492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [term 2 FOLLOWER]: Leader election won for term 2
I20260620 18:42:32.503904 13492 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [term 2 LEADER]: Becoming Leader. State: Replica: c2efcc70af1e4061b9d972d5b56de43d, State: Running, Role: LEADER
I20260620 18:42:32.504261 13492 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: false } } peers { permanent_uuid: "c2efcc70af1e4061b9d972d5b56de43d" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 45955 } attrs { promote: false } }
I20260620 18:42:32.506274 13502 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2efcc70af1e4061b9d972d5b56de43d. Latest consensus state: current_term: 2 leader_uuid: "c2efcc70af1e4061b9d972d5b56de43d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: false } } peers { permanent_uuid: "c2efcc70af1e4061b9d972d5b56de43d" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 45955 } attrs { promote: false } } }
I20260620 18:42:32.506657 13502 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:32.967451 13505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:32.966454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:34625 (local address 127.5.225.130:54871)
0620 18:42:32.967016 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:32.967118 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:34625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":306,"threads_started":1}
I20260620 18:42:32.970347 13486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:32.969861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:34625 (local address 127.5.225.131:44853)
0620 18:42:32.970032 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:32.970117 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:34625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 18:42:32.973197 13506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:32.972620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:34625 (local address 127.5.225.129:45271)
0620 18:42:32.972987 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:32.973059 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:34625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":195,"threads_started":1}
I20260620 18:42:33.019184 12770 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [term 2 FOLLOWER]: Refusing update from remote peer c2efcc70af1e4061b9d972d5b56de43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20260620 18:42:33.019853 13507 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 18:42:33.021600 13508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2efcc70af1e4061b9d972d5b56de43d. Latest consensus state: current_term: 2 leader_uuid: "c2efcc70af1e4061b9d972d5b56de43d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: false } } peers { permanent_uuid: "c2efcc70af1e4061b9d972d5b56de43d" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 45955 } attrs { promote: false } } }
I20260620 18:42:33.021878 13508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:42:33.024250 13507 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "c2efcc70af1e4061b9d972d5b56de43d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: false } } peers { permanent_uuid: "c2efcc70af1e4061b9d972d5b56de43d" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 45955 } attrs { promote: false } } }
I20260620 18:42:33.024616 13507 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:33.025727 13515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:33.025010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:34625 (local address 127.0.0.1:60556)
0620 18:42:33.025468 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:33.025554 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:34625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":247,"threads_started":1}
I20260620 18:42:33.026410 13508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "c2efcc70af1e4061b9d972d5b56de43d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6cf69d27a27142a9b0c07415bb23d249" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 34625 } } peers { permanent_uuid: "45a9ce1454f24d488dd7295235c75786" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 37095 } attrs { promote: false } } peers { permanent_uuid: "c2efcc70af1e4061b9d972d5b56de43d" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 45955 } attrs { promote: false } } }
I20260620 18:42:33.026754 13508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45a9ce1454f24d488dd7295235c75786 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:42:33.027748 13515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:33.027426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:34625 (local address 127.0.0.1:60566)
0620 18:42:33.027570 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:33.027635 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:34625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 18:42:33.027897 12936 consensus_peers.cc:487] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d -> Peer 6cf69d27a27142a9b0c07415bb23d249 (127.5.225.190:34625): Couldn't send request to peer 6cf69d27a27142a9b0c07415bb23d249. Status: Network error: Client connection negotiation failed: client connection to 127.5.225.190:34625: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 18:42:33.151974 13504 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:42:33.155669 13504 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:42:33.156627 13504 catalog_manager.cc:1259] Loaded cluster ID: fdd42e8c2ff34c219f495835c8ada190
I20260620 18:42:33.156736 13504 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:42:33.157867 13504 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:42:33.158716 13504 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d: Loaded TSK: 0
I20260620 18:42:33.159593 13504 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:42:33.195966 12950 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60462:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:42:33.202904 12950 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:42:33.235273 13519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:33.217388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.131:42693 (local address 127.0.0.1:45634)
0620 18:42:33.226928 (+ 9540us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:33.226959 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:33.227271 (+ 312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:33.228009 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:33.228018 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:33.228073 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:33.228533 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:33.228550 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:33.230093 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:33.230099 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:33.232341 (+ 2242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:33.232357 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:33.234341 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:33.234348 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:33.234537 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:33.234581 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 18:42:33.234687 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 18:42:33.234846 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9335,"thread_start_us":9093,"threads_started":1}
I20260620 18:42:33.235718 13518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:33.216736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56150 (local address 127.5.225.130:45529)
0620 18:42:33.229542 (+ 12806us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:33.229549 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:33.229568 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:42:33.229638 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:33.229644 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:33.229742 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:33.229847 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:33.230583 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:33.231756 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:33.233671 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:33.234520 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:33.234668 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:33.234772 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:33.235307 (+ 535us) server_negotiation.cc:299] Negotiation successful
0620 18:42:33.235441 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12630,"thread_start_us":12538,"threads_started":1}
I20260620 18:42:33.236598 13522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:33.218221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45634 (local address 127.5.225.131:42693)
0620 18:42:33.222670 (+ 4449us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:33.222684 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:33.227102 (+ 4418us) server_negotiation.cc:407] Connection header received
0620 18:42:33.227495 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:33.227505 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:33.227638 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:33.227736 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:33.228828 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:33.229958 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:33.232515 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:33.233940 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:33.234076 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:33.234235 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:33.235687 (+ 1452us) server_negotiation.cc:299] Negotiation successful
0620 18:42:33.236252 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4262,"thread_start_us":4115,"threads_started":1}
I20260620 18:42:33.235716 13515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:33.216644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.130:45529 (local address 127.0.0.1:56150)
0620 18:42:33.216813 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:33.216834 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:33.216950 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:33.229893 (+ 12943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:33.229900 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:33.229926 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:33.230363 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:33.230383 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:33.231927 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:33.231932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:33.233521 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:33.233532 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:33.234973 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:33.234983 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:33.235151 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:33.235189 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 18:42:33.235282 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 18:42:33.235421 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":1}
I20260620 18:42:33.237522 13521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:33.217870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59820 (local address 127.5.225.129:42219)
0620 18:42:33.218530 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:33.218541 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:33.229424 (+ 10883us) server_negotiation.cc:407] Connection header received
0620 18:42:33.229718 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:33.229723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:33.229787 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:33.229915 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:33.230906 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:33.232059 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:33.234783 (+ 2724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:33.236097 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:33.236209 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:33.236312 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:33.237001 (+ 689us) server_negotiation.cc:299] Negotiation successful
0620 18:42:33.237159 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":480,"threads_started":1}
I20260620 18:42:33.237939 13352 tablet_service.cc:1467] Processing CreateTablet for tablet efa7795baad54cfc937093b43e5f66e4 (DEFAULT_TABLE table=test-table [id=5ca875e78b31453cb40febf7cf11a766]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:42:33.237360 13520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:33.217412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:42219 (local address 127.0.0.1:59820)
0620 18:42:33.229307 (+ 11895us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:33.229335 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:33.229537 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:33.230137 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:33.230143 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:33.230195 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:33.230697 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:33.230710 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:33.232189 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:33.232198 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:33.234606 (+ 2408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:33.234621 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:33.236494 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:33.236500 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:33.236738 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:33.236773 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 18:42:33.236943 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 18:42:33.237089 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11742,"mutex_wait_us":25,"thread_start_us":11551,"threads_started":1}
I20260620 18:42:33.238458 13352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efa7795baad54cfc937093b43e5f66e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:33.238466 13211 tablet_service.cc:1467] Processing CreateTablet for tablet efa7795baad54cfc937093b43e5f66e4 (DEFAULT_TABLE table=test-table [id=5ca875e78b31453cb40febf7cf11a766]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:42:33.238891 13211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efa7795baad54cfc937093b43e5f66e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:33.241223 13063 tablet_service.cc:1467] Processing CreateTablet for tablet efa7795baad54cfc937093b43e5f66e4 (DEFAULT_TABLE table=test-table [id=5ca875e78b31453cb40febf7cf11a766]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:42:33.241700 13063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efa7795baad54cfc937093b43e5f66e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:33.244016 13523 tablet_bootstrap.cc:492] T efa7795baad54cfc937093b43e5f66e4 P 762e84cc86204351bc255b1e796e833f: Bootstrap starting.
I20260620 18:42:33.245563 13523 tablet_bootstrap.cc:654] T efa7795baad54cfc937093b43e5f66e4 P 762e84cc86204351bc255b1e796e833f: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:33.246905 13524 tablet_bootstrap.cc:492] T efa7795baad54cfc937093b43e5f66e4 P 3417abed2a0446919a8150bec84fbafc: Bootstrap starting.
I20260620 18:42:33.248528 13524 tablet_bootstrap.cc:654] T efa7795baad54cfc937093b43e5f66e4 P 3417abed2a0446919a8150bec84fbafc: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:33.248991 13523 tablet_bootstrap.cc:492] T efa7795baad54cfc937093b43e5f66e4 P 762e84cc86204351bc255b1e796e833f: No bootstrap required, opened a new log
I20260620 18:42:33.249164 13523 ts_tablet_manager.cc:1397] T efa7795baad54cfc937093b43e5f66e4 P 762e84cc86204351bc255b1e796e833f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 18:42:33.249502 13526 tablet_bootstrap.cc:492] T efa7795baad54cfc937093b43e5f66e4 P 911d8be242834719ac569e965601a2e2: Bootstrap starting.
I20260620 18:42:33.249994 13523 raft_consensus.cc:348] T efa7795baad54cfc937093b43e5f66e4 P 762e84cc86204351bc255b1e796e833f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762e84cc86204351bc255b1e796e833f" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 45529 } } peers { permanent_uuid: "911d8be242834719ac569e965601a2e2" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 42219 } } peers { permanent_uuid: "3417abed2a0446919a8150bec84fbafc" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 42693 } }
I20260620 18:42:33.250253 13523 raft_consensus.cc:374] T efa7795baad54cfc937093b43e5f66e4 P 762e84cc86204351bc255b1e796e833f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:33.250360 13523 raft_consensus.cc:729] T efa7795baad54cfc937093b43e5f66e4 P 762e84cc86204351bc255b1e796e833f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 762e84cc86204351bc255b1e796e833f, State: Initialized, Role: FOLLOWER
I20260620 18:42:33.250594 13523 consensus_queue.cc:260] T efa7795baad54cfc937093b43e5f66e4 P 762e84cc86204351bc255b1e796e833f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762e84cc86204351bc255b1e796e833f" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 45529 } } peers { permanent_uuid: "911d8be242834719ac569e965601a2e2" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 42219 } } peers { permanent_uuid: "3417abed2a0446919a8150bec84fbafc" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 42693 } }
I20260620 18:42:33.251134 13524 tablet_bootstrap.cc:492] T efa7795baad54cfc937093b43e5f66e4 P 3417abed2a0446919a8150bec84fbafc: No bootstrap required, opened a new log
I20260620 18:42:33.251294 13524 ts_tablet_manager.cc:1397] T efa7795baad54cfc937093b43e5f66e4 P 3417abed2a0446919a8150bec84fbafc: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 18:42:33.251395 13523 ts_tablet_manager.cc:1428] T efa7795baad54cfc937093b43e5f66e4 P 762e84cc86204351bc255b1e796e833f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 18:42:33.252406 13505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:33.251756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:34625 (local address 127.5.225.130:50013)
0620 18:42:33.252130 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:33.252223 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:34625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251}
I20260620 18:42:33.252382 13524 raft_consensus.cc:348] T efa7795baad54cfc937093b43e5f66e4 P 3417abed2a0446919a8150bec84fbafc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762e84cc86204351bc255b1e796e833f" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 45529 } } peers { permanent_uuid: "911d8be242834719ac569e965601a2e2" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 42219 } } peers { permanent_uuid: "3417abed2a0446919a8150bec84fbafc" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 42693 } }
I20260620 18:42:33.252656 13524 raft_consensus.cc:374] T efa7795baad54cfc937093b43e5f66e4 P 3417abed2a0446919a8150bec84fbafc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:33.252761 13524 raft_consensus.cc:729] T efa7795baad54cfc937093b43e5f66e4 P 3417abed2a0446919a8150bec84fbafc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3417abed2a0446919a8150bec84fbafc, State: Initialized, Role: FOLLOWER
I20260620 18:42:33.253034 13524 consensus_queue.cc:260] T efa7795baad54cfc937093b43e5f66e4 P 3417abed2a0446919a8150bec84fbafc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762e84cc86204351bc255b1e796e833f" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 45529 } } peers { permanent_uuid: "911d8be242834719ac569e965601a2e2" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 42219 } } peers { permanent_uuid: "3417abed2a0446919a8150bec84fbafc" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 42693 } }
I20260620 18:42:33.253758 13526 tablet_bootstrap.cc:654] T efa7795baad54cfc937093b43e5f66e4 P 911d8be242834719ac569e965601a2e2: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:33.253929 13524 ts_tablet_manager.cc:1428] T efa7795baad54cfc937093b43e5f66e4 P 3417abed2a0446919a8150bec84fbafc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:42:33.255723 13486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:33.255271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:34625 (local address 127.5.225.131:33257)
0620 18:42:33.255439 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:33.255543 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:34625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 18:42:33.256910 13277 heartbeater.cc:503] Master 127.5.225.188:45955 was elected leader, sending a full tablet report...
I20260620 18:42:33.257107 13486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:33.256623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:34625 (local address 127.5.225.131:46645)
0620 18:42:33.256867 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:33.256944 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:34625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260620 18:42:33.257120 13505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:33.256523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:34625 (local address 127.5.225.130:51289)
0620 18:42:33.256846 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:33.256962 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:34625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226}
I20260620 18:42:33.258556 13526 tablet_bootstrap.cc:492] T efa7795baad54cfc937093b43e5f66e4 P 911d8be242834719ac569e965601a2e2: No bootstrap required, opened a new log
I20260620 18:42:33.258724 13526 ts_tablet_manager.cc:1397] T efa7795baad54cfc937093b43e5f66e4 P 911d8be242834719ac569e965601a2e2: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260620 18:42:33.259466 13526 raft_consensus.cc:348] T efa7795baad54cfc937093b43e5f66e4 P 911d8be242834719ac569e965601a2e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762e84cc86204351bc255b1e796e833f" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 45529 } } peers { permanent_uuid: "911d8be242834719ac569e965601a2e2" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 42219 } } peers { permanent_uuid: "3417abed2a0446919a8150bec84fbafc" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 42693 } }
I20260620 18:42:33.259652 13526 raft_consensus.cc:374] T efa7795baad54cfc937093b43e5f66e4 P 911d8be242834719ac569e965601a2e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:33.259757 13526 raft_consensus.cc:729] T efa7795baad54cfc937093b43e5f66e4 P 911d8be242834719ac569e965601a2e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 911d8be242834719ac569e965601a2e2, State: Initialized, Role: FOLLOWER
I20260620 18:42:33.259984 13526 consensus_queue.cc:260] T efa7795baad54cfc937093b43e5f66e4 P 911d8be242834719ac569e965601a2e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762e84cc86204351bc255b1e796e833f" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 45529 } } peers { permanent_uuid: "911d8be242834719ac569e965601a2e2" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 42219 } } peers { permanent_uuid: "3417abed2a0446919a8150bec84fbafc" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 42693 } }
I20260620 18:42:33.260732 13526 ts_tablet_manager.cc:1428] T efa7795baad54cfc937093b43e5f66e4 P 911d8be242834719ac569e965601a2e2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:42:33.261575 13506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:33.261027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:34625 (local address 127.5.225.129:59169)
0620 18:42:33.261371 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:33.261462 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:34625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239}
I20260620 18:42:33.262423 13506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:33.262082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:34625 (local address 127.5.225.129:56593)
0620 18:42:33.262237 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:33.262308 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:34625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 18:42:33.263437 13129 heartbeater.cc:503] Master 127.5.225.188:45955 was elected leader, sending a full tablet report...
I20260620 18:42:33.268365 13529 raft_consensus.cc:482] T efa7795baad54cfc937093b43e5f66e4 P 3417abed2a0446919a8150bec84fbafc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:42:33.268601 13529 raft_consensus.cc:504] T efa7795baad54cfc937093b43e5f66e4 P 3417abed2a0446919a8150bec84fbafc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762e84cc86204351bc255b1e796e833f" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 45529 } } peers { permanent_uuid: "911d8be242834719ac569e965601a2e2" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 42219 } } peers { permanent_uuid: "3417abed2a0446919a8150bec84fbafc" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 42693 } }
I20260620 18:42:33.269272 13529 leader_election.cc:288] T efa7795baad54cfc937093b43e5f66e4 P 3417abed2a0446919a8150bec84fbafc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 762e84cc86204351bc255b1e796e833f (127.5.225.130:45529), 911d8be242834719ac569e965601a2e2 (127.5.225.129:42219)
I20260620 18:42:33.269773 13083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7795baad54cfc937093b43e5f66e4" candidate_uuid: "3417abed2a0446919a8150bec84fbafc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "911d8be242834719ac569e965601a2e2" is_pre_election: true
I20260620 18:42:33.269774 13231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7795baad54cfc937093b43e5f66e4" candidate_uuid: "3417abed2a0446919a8150bec84fbafc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "762e84cc86204351bc255b1e796e833f" is_pre_election: true
I20260620 18:42:33.270028 13083 raft_consensus.cc:2454] T efa7795baad54cfc937093b43e5f66e4 P 911d8be242834719ac569e965601a2e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3417abed2a0446919a8150bec84fbafc in term 0.
I20260620 18:42:33.270232 13231 raft_consensus.cc:2454] T efa7795baad54cfc937093b43e5f66e4 P 762e84cc86204351bc255b1e796e833f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3417abed2a0446919a8150bec84fbafc in term 0.
I20260620 18:42:33.270454 13306 leader_election.cc:302] T efa7795baad54cfc937093b43e5f66e4 P 3417abed2a0446919a8150bec84fbafc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3417abed2a0446919a8150bec84fbafc, 911d8be242834719ac569e965601a2e2; no voters:
I20260620 18:42:33.270721 13529 raft_consensus.cc:2781] T efa7795baad54cfc937093b43e5f66e4 P 3417abed2a0446919a8150bec84fbafc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:42:33.270845 13529 raft_consensus.cc:482] T efa7795baad54cfc937093b43e5f66e4 P 3417abed2a0446919a8150bec84fbafc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:42:33.270936 13529 raft_consensus.cc:3037] T efa7795baad54cfc937093b43e5f66e4 P 3417abed2a0446919a8150bec84fbafc [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:33.273047 13529 raft_consensus.cc:504] T efa7795baad54cfc937093b43e5f66e4 P 3417abed2a0446919a8150bec84fbafc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762e84cc86204351bc255b1e796e833f" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 45529 } } peers { permanent_uuid: "911d8be242834719ac569e965601a2e2" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 42219 } } peers { permanent_uuid: "3417abed2a0446919a8150bec84fbafc" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 42693 } }
I20260620 18:42:33.273584 13529 leader_election.cc:288] T efa7795baad54cfc937093b43e5f66e4 P 3417abed2a0446919a8150bec84fbafc [CANDIDATE]: Term 1 election: Requested vote from peers 762e84cc86204351bc255b1e796e833f (127.5.225.130:45529), 911d8be242834719ac569e965601a2e2 (127.5.225.129:42219)
I20260620 18:42:33.274098 13231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7795baad54cfc937093b43e5f66e4" candidate_uuid: "3417abed2a0446919a8150bec84fbafc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "762e84cc86204351bc255b1e796e833f"
I20260620 18:42:33.274096 13083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7795baad54cfc937093b43e5f66e4" candidate_uuid: "3417abed2a0446919a8150bec84fbafc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "911d8be242834719ac569e965601a2e2"
I20260620 18:42:33.274343 13231 raft_consensus.cc:3037] T efa7795baad54cfc937093b43e5f66e4 P 762e84cc86204351bc255b1e796e833f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:33.274343 13083 raft_consensus.cc:3037] T efa7795baad54cfc937093b43e5f66e4 P 911d8be242834719ac569e965601a2e2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:33.277415 13231 raft_consensus.cc:2454] T efa7795baad54cfc937093b43e5f66e4 P 762e84cc86204351bc255b1e796e833f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3417abed2a0446919a8150bec84fbafc in term 1.
I20260620 18:42:33.277460 13083 raft_consensus.cc:2454] T efa7795baad54cfc937093b43e5f66e4 P 911d8be242834719ac569e965601a2e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3417abed2a0446919a8150bec84fbafc in term 1.
I20260620 18:42:33.277877 13305 leader_election.cc:302] T efa7795baad54cfc937093b43e5f66e4 P 3417abed2a0446919a8150bec84fbafc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3417abed2a0446919a8150bec84fbafc, 762e84cc86204351bc255b1e796e833f; no voters:
I20260620 18:42:33.278223 13529 raft_consensus.cc:2781] T efa7795baad54cfc937093b43e5f66e4 P 3417abed2a0446919a8150bec84fbafc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:42:33.278741 13529 raft_consensus.cc:686] T efa7795baad54cfc937093b43e5f66e4 P 3417abed2a0446919a8150bec84fbafc [term 1 LEADER]: Becoming Leader. State: Replica: 3417abed2a0446919a8150bec84fbafc, State: Running, Role: LEADER
I20260620 18:42:33.279289 13529 consensus_queue.cc:237] T efa7795baad54cfc937093b43e5f66e4 P 3417abed2a0446919a8150bec84fbafc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762e84cc86204351bc255b1e796e833f" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 45529 } } peers { permanent_uuid: "911d8be242834719ac569e965601a2e2" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 42219 } } peers { permanent_uuid: "3417abed2a0446919a8150bec84fbafc" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 42693 } }
I20260620 18:42:33.280488 13418 heartbeater.cc:503] Master 127.5.225.188:45955 was elected leader, sending a full tablet report...
I20260620 18:42:33.281654 13486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:33.281055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:34625 (local address 127.5.225.131:46383)
0620 18:42:33.281361 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:33.281484 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:34625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186}
I20260620 18:42:33.283511 12950 catalog_manager.cc:5495] T efa7795baad54cfc937093b43e5f66e4 P 3417abed2a0446919a8150bec84fbafc reported cstate change: term changed from 0 to 1, leader changed from <none> to 3417abed2a0446919a8150bec84fbafc (127.5.225.131). New cstate: current_term: 1 leader_uuid: "3417abed2a0446919a8150bec84fbafc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762e84cc86204351bc255b1e796e833f" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 45529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "911d8be242834719ac569e965601a2e2" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 42219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3417abed2a0446919a8150bec84fbafc" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 42693 } health_report { overall_health: HEALTHY } } }
I20260620 18:42:33.289196 13352 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e7a18e9cbb04e2e9d0690ae3bcefcc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:42:30 UTC) from {username='oryx', principal='oryx/127.5.225.188@KRBTEST.COM'} at 127.0.0.1:45634
I20260620 18:42:33.290130 13534 tablet_replica.cc:330] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 3417abed2a0446919a8150bec84fbafc: stopping tablet replica
I20260620 18:42:33.290668 13534 raft_consensus.cc:2229] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 3417abed2a0446919a8150bec84fbafc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:42:33.291209 13534 raft_consensus.cc:2258] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 3417abed2a0446919a8150bec84fbafc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:42:33.292810 13534 ts_tablet_manager.cc:1905] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 3417abed2a0446919a8150bec84fbafc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:42:33.299575 13534 ts_tablet_manager.cc:1918] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 3417abed2a0446919a8150bec84fbafc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:42:33.299768 13534 log.cc:1196] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 3417abed2a0446919a8150bec84fbafc: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-2/wal/wals/8e7a18e9cbb04e2e9d0690ae3bcefcc3
I20260620 18:42:33.300364 13534 ts_tablet_manager.cc:1939] T 8e7a18e9cbb04e2e9d0690ae3bcefcc3 P 3417abed2a0446919a8150bec84fbafc: Deleting consensus metadata
I20260620 18:42:33.301898 12936 catalog_manager.cc:4841] TS 3417abed2a0446919a8150bec84fbafc (127.5.225.131:42693): tablet 8e7a18e9cbb04e2e9d0690ae3bcefcc3 (table test-table [id=628e1f9654bc467aa640c341b8c79c67]) successfully deleted
I20260620 18:42:33.304000 13522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:33.298422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45648 (local address 127.5.225.131:42693)
0620 18:42:33.298629 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:33.298635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:33.298655 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:42:33.298803 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:33.298808 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:33.298900 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:33.299039 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:42:33.299411 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:33.300584 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:33.301990 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:33.302478 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:33.302575 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:33.302960 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:42:33.303492 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:42:33.303599 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:33.303666 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 18:42:33.303730 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 18:42:33.307940 13083 raft_consensus.cc:1261] T efa7795baad54cfc937093b43e5f66e4 P 911d8be242834719ac569e965601a2e2 [term 1 FOLLOWER]: Refusing update from remote peer 3417abed2a0446919a8150bec84fbafc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:42:33.308212 13231 raft_consensus.cc:1261] T efa7795baad54cfc937093b43e5f66e4 P 762e84cc86204351bc255b1e796e833f [term 1 FOLLOWER]: Refusing update from remote peer 3417abed2a0446919a8150bec84fbafc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:42:33.308598 13532 consensus_queue.cc:1035] T efa7795baad54cfc937093b43e5f66e4 P 3417abed2a0446919a8150bec84fbafc [LEADER]: Connected to new peer: Peer: permanent_uuid: "911d8be242834719ac569e965601a2e2" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 42219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:42:33.308961 13529 consensus_queue.cc:1035] T efa7795baad54cfc937093b43e5f66e4 P 3417abed2a0446919a8150bec84fbafc [LEADER]: Connected to new peer: Peer: permanent_uuid: "762e84cc86204351bc255b1e796e833f" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 45529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:42:33.311861 13506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:33.311210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:34625 (local address 127.5.225.129:46087)
0620 18:42:33.311495 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:33.311622 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:34625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260620 18:42:33.312436 13505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:33.311209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:34625 (local address 127.5.225.130:41163)
0620 18:42:33.312108 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:33.312239 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:34625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":762}
I20260620 18:42:33.324888 13538 mvcc.cc:205] Tried to move back new op lower bound from 7298993984742731776 to 7298993984635871232. Current Snapshot: MvccSnapshot[applied={T|T < 7298993984742731776}]
I20260620 18:42:33.326227 13506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:33.325560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:34625 (local address 127.5.225.129:43419)
0620 18:42:33.325836 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:33.325952 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:34625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260620 18:42:33.326612 13486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:33.325547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:34625 (local address 127.5.225.131:35309)
0620 18:42:33.326326 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:33.326441 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:34625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":671}
I20260620 18:42:33.326817 13505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:33.326368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:34625 (local address 127.5.225.130:41085)
0620 18:42:33.326571 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:33.326664 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:34625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260620 18:42:33.336545 12949 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:42:33.336922 12949 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:42:33.342957 12949 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c2efcc70af1e4061b9d972d5b56de43d: Sending DeleteTablet for 3 replicas of tablet efa7795baad54cfc937093b43e5f66e4
I20260620 18:42:33.343909 13063 tablet_service.cc:1514] Processing DeleteTablet for tablet efa7795baad54cfc937093b43e5f66e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:42:33 UTC) from {username='oryx', principal='oryx/127.5.225.188@KRBTEST.COM'} at 127.0.0.1:59820
I20260620 18:42:33.343912 13352 tablet_service.cc:1514] Processing DeleteTablet for tablet efa7795baad54cfc937093b43e5f66e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:42:33 UTC) from {username='oryx', principal='oryx/127.5.225.188@KRBTEST.COM'} at 127.0.0.1:45634
I20260620 18:42:33.343940 13211 tablet_service.cc:1514] Processing DeleteTablet for tablet efa7795baad54cfc937093b43e5f66e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:42:33 UTC) from {username='oryx', principal='oryx/127.5.225.188@KRBTEST.COM'} at 127.0.0.1:56150
I20260620 18:42:33.344189 13534 tablet_replica.cc:330] T efa7795baad54cfc937093b43e5f66e4 P 3417abed2a0446919a8150bec84fbafc: stopping tablet replica
I20260620 18:42:33.344486 13534 raft_consensus.cc:2229] T efa7795baad54cfc937093b43e5f66e4 P 3417abed2a0446919a8150bec84fbafc [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:42:33.344877 13545 tablet_replica.cc:330] T efa7795baad54cfc937093b43e5f66e4 P 762e84cc86204351bc255b1e796e833f: stopping tablet replica
I20260620 18:42:33.344974 13534 raft_consensus.cc:2258] T efa7795baad54cfc937093b43e5f66e4 P 3417abed2a0446919a8150bec84fbafc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:42:33.345207 13545 raft_consensus.cc:2229] T efa7795baad54cfc937093b43e5f66e4 P 762e84cc86204351bc255b1e796e833f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:42:33.345578 13545 raft_consensus.cc:2258] T efa7795baad54cfc937093b43e5f66e4 P 762e84cc86204351bc255b1e796e833f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:42:33.346181 13534 ts_tablet_manager.cc:1905] T efa7795baad54cfc937093b43e5f66e4 P 3417abed2a0446919a8150bec84fbafc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:42:33.346477 13546 tablet_replica.cc:330] T efa7795baad54cfc937093b43e5f66e4 P 911d8be242834719ac569e965601a2e2: stopping tablet replica
I20260620 18:42:33.346532 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 12922
I20260620 18:42:33.346818 13545 ts_tablet_manager.cc:1905] T efa7795baad54cfc937093b43e5f66e4 P 762e84cc86204351bc255b1e796e833f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:42:33.346776 13546 raft_consensus.cc:2229] T efa7795baad54cfc937093b43e5f66e4 P 911d8be242834719ac569e965601a2e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:42:33.347249 13546 raft_consensus.cc:2258] T efa7795baad54cfc937093b43e5f66e4 P 911d8be242834719ac569e965601a2e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:42:33.352217 13534 ts_tablet_manager.cc:1918] T efa7795baad54cfc937093b43e5f66e4 P 3417abed2a0446919a8150bec84fbafc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:42:33.352383 13534 log.cc:1196] T efa7795baad54cfc937093b43e5f66e4 P 3417abed2a0446919a8150bec84fbafc: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-2/wal/wals/efa7795baad54cfc937093b43e5f66e4
I20260620 18:42:33.352830 13534 ts_tablet_manager.cc:1939] T efa7795baad54cfc937093b43e5f66e4 P 3417abed2a0446919a8150bec84fbafc: Deleting consensus metadata
I20260620 18:42:33.353004 13545 ts_tablet_manager.cc:1918] T efa7795baad54cfc937093b43e5f66e4 P 762e84cc86204351bc255b1e796e833f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:42:33.353121 13545 log.cc:1196] T efa7795baad54cfc937093b43e5f66e4 P 762e84cc86204351bc255b1e796e833f: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-1/wal/wals/efa7795baad54cfc937093b43e5f66e4
I20260620 18:42:33.353588 13545 ts_tablet_manager.cc:1939] T efa7795baad54cfc937093b43e5f66e4 P 762e84cc86204351bc255b1e796e833f: Deleting consensus metadata
I20260620 18:42:33.353811 12936 catalog_manager.cc:4841] TS 3417abed2a0446919a8150bec84fbafc (127.5.225.131:42693): tablet efa7795baad54cfc937093b43e5f66e4 (table test-table [id=5ca875e78b31453cb40febf7cf11a766]) successfully deleted
I20260620 18:42:33.354660 12934 catalog_manager.cc:4841] TS 762e84cc86204351bc255b1e796e833f (127.5.225.130:45529): tablet efa7795baad54cfc937093b43e5f66e4 (table test-table [id=5ca875e78b31453cb40febf7cf11a766]) successfully deleted
I20260620 18:42:33.358729 13546 ts_tablet_manager.cc:1905] T efa7795baad54cfc937093b43e5f66e4 P 911d8be242834719ac569e965601a2e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:42:33.365141 13546 ts_tablet_manager.cc:1918] T efa7795baad54cfc937093b43e5f66e4 P 911d8be242834719ac569e965601a2e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:42:33.365293 13546 log.cc:1196] T efa7795baad54cfc937093b43e5f66e4 P 911d8be242834719ac569e965601a2e2: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980894474392-6022-0/minicluster-data/ts-0/wal/wals/efa7795baad54cfc937093b43e5f66e4
I20260620 18:42:33.365801 13546 ts_tablet_manager.cc:1939] T efa7795baad54cfc937093b43e5f66e4 P 911d8be242834719ac569e965601a2e2: Deleting consensus metadata
I20260620 18:42:33.367000 12935 catalog_manager.cc:4841] TS 911d8be242834719ac569e965601a2e2 (127.5.225.129:42219): tablet efa7795baad54cfc937093b43e5f66e4 (table test-table [id=5ca875e78b31453cb40febf7cf11a766]) successfully deleted
I20260620 18:42:33.367321 13521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:33.347887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59826 (local address 127.5.225.129:42219)
0620 18:42:33.348198 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:33.348204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:33.350727 (+ 2523us) server_negotiation.cc:407] Connection header received
0620 18:42:33.350899 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:33.350904 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:33.350998 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:33.351102 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:33.352917 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:33.354021 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:33.355803 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:33.356182 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:33.356285 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:33.359616 (+ 3331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:33.359634 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:33.359655 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:33.359700 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:33.363955 (+ 4255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:33.364799 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:33.364814 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:33.364818 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:33.364976 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:33.365440 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:33.365446 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:33.365454 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:33.365961 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:33.366289 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:33.366597 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 18:42:33.366839 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260620 18:42:33.367972 13123 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:33.507308 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 12922
I20260620 18:42:33.525542 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 13140
I20260620 18:42:33.541751 13518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:33.526533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56166 (local address 127.5.225.130:45529)
0620 18:42:33.526728 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:33.526732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:33.526770 (+ 38us) server_negotiation.cc:407] Connection header received
0620 18:42:33.526995 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:33.527036 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:33.527134 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:33.527289 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:33.529023 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:33.529786 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:33.531646 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:33.532071 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:33.532176 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:33.535544 (+ 3368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:33.535561 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:33.535577 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:33.535614 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:33.538717 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:33.539598 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:33.539605 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:33.539608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:33.539743 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:33.540120 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:33.540123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:33.540125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:33.540483 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:33.540687 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:33.541181 (+ 494us) server_negotiation.cc:299] Negotiation successful
0620 18:42:33.541351 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 18:42:33.542551 13271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:33.632218 13520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:33.631688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:34625 (local address 127.0.0.1:60590)
0620 18:42:33.631903 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:33.632029 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:34625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 18:42:33.633618 13520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:33.633243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:34625 (local address 127.0.0.1:60604)
0620 18:42:33.633391 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:33.633473 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:34625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 18:42:33.661782 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 13140
I20260620 18:42:33.688786 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 13289
I20260620 18:42:33.705056 13522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:33.689550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45660 (local address 127.5.225.131:42693)
0620 18:42:33.689721 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:33.689725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:33.689808 (+ 83us) server_negotiation.cc:407] Connection header received
0620 18:42:33.690015 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:33.690035 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:33.690147 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:33.690294 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:33.692128 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:33.692931 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:33.695002 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:33.695496 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:33.695606 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:33.698976 (+ 3370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:33.698988 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:33.699005 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:33.699040 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:33.702254 (+ 3214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:33.703062 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:33.703069 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:33.703072 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:33.703241 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:33.703635 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:33.703638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:33.703640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:33.703987 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:33.704237 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:33.704564 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 18:42:33.704728 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:42:33.705752 13412 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:33.844069 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 13289
I20260620 18:42:33.868005 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 12624
I20260620 18:42:33.869078 12791 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:33.983322 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 12624
I20260620 18:42:34.004175 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 12724
I20260620 18:42:34.005250 12980 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:34.123077 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 12724
2026-06-20T18:42:34Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10310 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:42:34 dist-test-slave-k6lg krb5kdc[13556](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:42:34 dist-test-slave-k6lg krb5kdc[13556](info): set up 2 sockets
Jun 20 18:42:34 dist-test-slave-k6lg krb5kdc[13556](info): commencing operation
krb5kdc: starting...
W20260620 18:42:35.219002 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:42:35 dist-test-slave-k6lg krb5kdc[13556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:42:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:42:35Z Disabled control of system clock
WARNING: no policy specified for oryx/127.5.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.190.keytab.
Entry for principal oryx/127.5.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.190.keytab.
I20260620 18:42:35.426752 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:40433
--webserver_interface=127.5.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.190.keytab
--principal=oryx/127.5.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:45741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:40433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:35.741094 13569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:35.741375 13569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:35.741462 13569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:35.748694 13569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:42:35.748796 13569 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 18:42:35.748836 13569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:35.748898 13569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:35.748946 13569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:42:35.749004 13569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:42:35.758706 13569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:45741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:40433
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.190.keytab
--principal=oryx/127.5.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:40433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:35.759948 13569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:35.761207 13569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:35.772737 13574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:35.772794 13577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:35.772797 13575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:35.773690 13569 server_base.cc:1034] running on GCE node
I20260620 18:42:35.774477 13569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:35.775832 13569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:35.777035 13569 hybrid_clock.cc:648] HybridClock initialized: now 1781980955776980 us; error 32 us; skew 500 ppm
I20260620 18:42:35.777346 13569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:35 dist-test-slave-k6lg krb5kdc[13556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980955, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:35.780284 13569 init.cc:377] Logged in from keytab as oryx/127.5.225.190@KRBTEST.COM (short username oryx)
I20260620 18:42:35.781559 13569 webserver.cc:460] Webserver started at http://127.5.225.190:41989/ using document root <none> and password file <none>
I20260620 18:42:35.782253 13569 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:35.782343 13569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:35.782619 13569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:35.785442 13569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "972f20065ea9422a838127c325ed3908"
format_stamp: "Formatted at 2026-06-20 18:42:35 on dist-test-slave-k6lg"
server_key: "711fdaeb1696a1651e740625a02c4256"
server_key_iv: "0759a1ae071f62686e58ab4d6f8bff1e"
server_key_version: "encryptionkey@0"
I20260620 18:42:35.786201 13569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "972f20065ea9422a838127c325ed3908"
format_stamp: "Formatted at 2026-06-20 18:42:35 on dist-test-slave-k6lg"
server_key: "711fdaeb1696a1651e740625a02c4256"
server_key_iv: "0759a1ae071f62686e58ab4d6f8bff1e"
server_key_version: "encryptionkey@0"
I20260620 18:42:35.790990 13569 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 18:42:35.794379 13584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:35.795943 13569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:42:35.796090 13569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "972f20065ea9422a838127c325ed3908"
format_stamp: "Formatted at 2026-06-20 18:42:35 on dist-test-slave-k6lg"
server_key: "711fdaeb1696a1651e740625a02c4256"
server_key_iv: "0759a1ae071f62686e58ab4d6f8bff1e"
server_key_version: "encryptionkey@0"
I20260620 18:42:35.796265 13569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:35.812310 13569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:35.815959 13569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:35.816157 13569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:35.837350 13569 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:40433
I20260620 18:42:35.837379 13645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:40433 every 8 connection(s)
I20260620 18:42:35.838737 13569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:42:35.842581 13646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:35.846065 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 13569
I20260620 18:42:35.846302 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:42:35.846935 6022 external_mini_cluster.cc:1428] Setting key 5b35f0c13cbc8b4f345e2c0f8a06687c
I20260620 18:42:35.854933 13646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 972f20065ea9422a838127c325ed3908: Bootstrap starting.
I20260620 18:42:35.858848 13646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 972f20065ea9422a838127c325ed3908: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:35.860180 13646 log.cc:824] T 00000000000000000000000000000000 P 972f20065ea9422a838127c325ed3908: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:35.863335 13646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 972f20065ea9422a838127c325ed3908: No bootstrap required, opened a new log
Jun 20 18:42:35 dist-test-slave-k6lg krb5kdc[13556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM
I20260620 18:42:35.872903 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:35.848899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35404 (local address 127.5.225.190:40433)
0620 18:42:35.849707 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:35.849722 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:35.849789 (+ 67us) server_negotiation.cc:407] Connection header received
0620 18:42:35.853192 (+ 3403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:35.853223 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:35.853757 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:35.854227 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:35.855567 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:35.857622 (+ 2055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:35.858885 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:35.859929 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:35.860125 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:35.864447 (+ 4322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:35.864458 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:35.864476 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:35.864546 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:35.868470 (+ 3924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:35.869181 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:35.869189 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:35.869197 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:35.869396 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:35.869750 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:35.869755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:35.869759 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:35.870488 (+ 729us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:35.870799 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:35.871109 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 18:42:35.871521 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":324,"threads_started":1}
I20260620 18:42:35.874135 13646 raft_consensus.cc:348] T 00000000000000000000000000000000 P 972f20065ea9422a838127c325ed3908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "972f20065ea9422a838127c325ed3908" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 40433 } }
I20260620 18:42:35.874567 13646 raft_consensus.cc:374] T 00000000000000000000000000000000 P 972f20065ea9422a838127c325ed3908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:35.874681 13646 raft_consensus.cc:729] T 00000000000000000000000000000000 P 972f20065ea9422a838127c325ed3908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 972f20065ea9422a838127c325ed3908, State: Initialized, Role: FOLLOWER
I20260620 18:42:35.875303 13646 consensus_queue.cc:260] T 00000000000000000000000000000000 P 972f20065ea9422a838127c325ed3908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "972f20065ea9422a838127c325ed3908" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 40433 } }
I20260620 18:42:35.875559 13646 raft_consensus.cc:388] T 00000000000000000000000000000000 P 972f20065ea9422a838127c325ed3908 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:42:35.875691 13646 raft_consensus.cc:482] T 00000000000000000000000000000000 P 972f20065ea9422a838127c325ed3908 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:42:35.875880 13646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 972f20065ea9422a838127c325ed3908 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:35.877947 13646 raft_consensus.cc:504] T 00000000000000000000000000000000 P 972f20065ea9422a838127c325ed3908 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "972f20065ea9422a838127c325ed3908" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 40433 } }
I20260620 18:42:35.878459 13646 leader_election.cc:302] T 00000000000000000000000000000000 P 972f20065ea9422a838127c325ed3908 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 972f20065ea9422a838127c325ed3908; no voters:
I20260620 18:42:35.878994 13646 leader_election.cc:288] T 00000000000000000000000000000000 P 972f20065ea9422a838127c325ed3908 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:42:35.879144 13651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 972f20065ea9422a838127c325ed3908 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:42:35.879803 13651 raft_consensus.cc:686] T 00000000000000000000000000000000 P 972f20065ea9422a838127c325ed3908 [term 1 LEADER]: Becoming Leader. State: Replica: 972f20065ea9422a838127c325ed3908, State: Running, Role: LEADER
I20260620 18:42:35.880275 13651 consensus_queue.cc:237] T 00000000000000000000000000000000 P 972f20065ea9422a838127c325ed3908 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "972f20065ea9422a838127c325ed3908" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 40433 } }
I20260620 18:42:35.881223 13646 sys_catalog.cc:564] T 00000000000000000000000000000000 P 972f20065ea9422a838127c325ed3908 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:42:35.883244 13653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 972f20065ea9422a838127c325ed3908 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 972f20065ea9422a838127c325ed3908. Latest consensus state: current_term: 1 leader_uuid: "972f20065ea9422a838127c325ed3908" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "972f20065ea9422a838127c325ed3908" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 40433 } } }
I20260620 18:42:35.883209 13652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 972f20065ea9422a838127c325ed3908 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "972f20065ea9422a838127c325ed3908" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "972f20065ea9422a838127c325ed3908" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 40433 } } }
I20260620 18:42:35.883693 13653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 972f20065ea9422a838127c325ed3908 [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:35.883843 13652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 972f20065ea9422a838127c325ed3908 [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:35.888296 13658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:42:35.893219 13658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:42:35.902616 13658 catalog_manager.cc:1347] Generated new cluster ID: d5d96ab326164694bbefc58b9e21890f
I20260620 18:42:35.902729 13658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:42:35.926522 13658 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:42:35.927899 13658 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:42:35.935411 13658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 972f20065ea9422a838127c325ed3908: Generated new TSK 0
I20260620 18:42:35.936229 13658 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.5.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.129.keytab.
Entry for principal oryx/127.5.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.129.keytab.
I20260620 18:42:35.989420 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.129:0
--local_ip_for_outbound_sockets=127.5.225.129
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:40433
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.129.keytab
--principal=oryx/127.5.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:45741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:36.311625 13672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:36.311895 13672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:36.312072 13672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:36.319343 13672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:36.319475 13672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:36.319624 13672 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:42:36.319748 13672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.129
I20260620 18:42:36.329468 13672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:45741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.129.keytab
--principal=oryx/127.5.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.225.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.225.190:40433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.129
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:36.330893 13672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:36.332250 13672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:36.345038 13677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:36.345095 13680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:36.345124 13678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:36.345865 13672 server_base.cc:1034] running on GCE node
I20260620 18:42:36.346499 13672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:36.347398 13672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:36.348649 13672 hybrid_clock.cc:648] HybridClock initialized: now 1781980956348574 us; error 50 us; skew 500 ppm
I20260620 18:42:36.348958 13672 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:36 dist-test-slave-k6lg krb5kdc[13556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980956, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:36.352146 13672 init.cc:377] Logged in from keytab as oryx/127.5.225.129@KRBTEST.COM (short username oryx)
I20260620 18:42:36.353561 13672 webserver.cc:460] Webserver started at http://127.5.225.129:43989/ using document root <none> and password file <none>
I20260620 18:42:36.354271 13672 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:36.354362 13672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:36.354626 13672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:36.357456 13672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-0/data/instance:
uuid: "06416c21938742de87f05ca67fcc42dc"
format_stamp: "Formatted at 2026-06-20 18:42:36 on dist-test-slave-k6lg"
server_key: "654843ce37694abc8d70eb26b440e924"
server_key_iv: "9006d9497962c3c07f281a5926bc7c63"
server_key_version: "encryptionkey@0"
I20260620 18:42:36.358260 13672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance:
uuid: "06416c21938742de87f05ca67fcc42dc"
format_stamp: "Formatted at 2026-06-20 18:42:36 on dist-test-slave-k6lg"
server_key: "654843ce37694abc8d70eb26b440e924"
server_key_iv: "9006d9497962c3c07f281a5926bc7c63"
server_key_version: "encryptionkey@0"
I20260620 18:42:36.363410 13672 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:42:36.366643 13687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:36.367951 13672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:42:36.368132 13672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-0/wal
uuid: "06416c21938742de87f05ca67fcc42dc"
format_stamp: "Formatted at 2026-06-20 18:42:36 on dist-test-slave-k6lg"
server_key: "654843ce37694abc8d70eb26b440e924"
server_key_iv: "9006d9497962c3c07f281a5926bc7c63"
server_key_version: "encryptionkey@0"
I20260620 18:42:36.368356 13672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:36.383106 13672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:36.387364 13672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:36.387635 13672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:36.390322 13672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:36.390426 13672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:36.390595 13672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:36.390673 13672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:42:36 dist-test-slave-k6lg krb5kdc[13556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980956, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.225.129@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM
I20260620 18:42:36.414983 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:36.390939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:58979 (local address 127.5.225.190:40433)
0620 18:42:36.393399 (+ 2460us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:36.393406 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:36.393431 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:42:36.395142 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:36.395148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:36.395248 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:36.395374 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:36.397607 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:36.398699 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:36.401502 (+ 2803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:36.402266 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:36.402429 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:36.407715 (+ 5286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:36.407720 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:36.407723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:36.407756 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:36.411287 (+ 3531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:36.412222 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:36.412230 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:36.412235 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:36.412411 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:36.412867 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:36.412874 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:36.412881 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:36.413318 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:36.413489 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:36.414178 (+ 689us) server_negotiation.cc:299] Negotiation successful
0620 18:42:36.414439 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2329,"thread_start_us":2223,"threads_started":1}
I20260620 18:42:36.415840 13697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:36.391657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:40433 (local address 127.5.225.129:58979)
0620 18:42:36.392704 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:36.392767 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:36.394859 (+ 2092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:36.395611 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:36.395618 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:36.396448 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:36.397385 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:36.397419 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:36.398851 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:36.398857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:36.401336 (+ 2479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:36.401355 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:36.402790 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:36.402798 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:36.403174 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:36.404111 (+ 937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:36.404152 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:36.407486 (+ 3334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:36.411481 (+ 3995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:36.411493 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:36.411516 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:36.412086 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:36.412535 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:36.412539 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:36.412543 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:36.412761 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:36.413449 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:36.413456 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:36.413790 (+ 334us) client_negotiation.cc:769] Sending connection context
0620 18:42:36.414023 (+ 233us) client_negotiation.cc:240] Negotiation successful
0620 18:42:36.414631 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":395,"threads_started":1}
I20260620 18:42:36.436916 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:36.421497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:50409 (local address 127.5.225.190:40433)
0620 18:42:36.422122 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:36.422132 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:36.422490 (+ 358us) server_negotiation.cc:407] Connection header received
0620 18:42:36.422682 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:36.422689 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:36.422757 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:36.422853 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:36.424061 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:36.425101 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:36.427266 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:36.427897 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:36.428067 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:36.430834 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:36.430843 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:36.430849 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:36.430892 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:36.433884 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:36.434806 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:36.434811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:36.434815 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:36.434949 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:36.435326 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:36.435332 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:36.435335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:36.435732 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:36.435886 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:36.436314 (+ 428us) server_negotiation.cc:299] Negotiation successful
0620 18:42:36.436543 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468}
I20260620 18:42:36.437028 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:36.421624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:40433 (local address 127.5.225.129:50409)
0620 18:42:36.422391 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:36.422406 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:36.422562 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:36.422878 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:36.422883 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:36.423381 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:36.423855 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:36.423870 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:36.425265 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:36.425270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:36.427105 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:36.427126 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:36.428136 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:36.428150 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:36.428406 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:36.429184 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:36.429216 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:36.430594 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:36.434017 (+ 3423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:36.434022 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:36.434027 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:36.434660 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:36.435057 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:36.435062 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:36.435066 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:36.435206 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:36.435862 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:36.435867 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:36.436049 (+ 182us) client_negotiation.cc:769] Sending connection context
0620 18:42:36.436185 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 18:42:36.436549 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":517,"threads_started":1}
I20260620 18:42:36.446504 13672 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.129:44023
I20260620 18:42:36.446532 13807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.129:44023 every 8 connection(s)
I20260620 18:42:36.448020 13672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
I20260620 18:42:36.449553 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 13672
I20260620 18:42:36.449757 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance
I20260620 18:42:36.450390 13808 heartbeater.cc:347] Connected to a master server at 127.5.225.190:40433
I20260620 18:42:36.450609 13808 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:36.450639 6022 external_mini_cluster.cc:1428] Setting key 4f6269e41d436096a75ac10c9e6ac30e
I20260620 18:42:36.451421 13808 heartbeater.cc:511] Master 127.5.225.190:40433 requested a full tablet report, sending...
I20260620 18:42:36.453331 13601 ts_manager.cc:194] Registered new tserver with Master: 06416c21938742de87f05ca67fcc42dc (127.5.225.129:44023)
I20260620 18:42:36.454857 13601 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.225.129@KRBTEST.COM'} at 127.5.225.129:58979
WARNING: no policy specified for oryx/127.5.225.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.225.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.130.keytab.
Entry for principal oryx/127.5.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.130.keytab.
I20260620 18:42:36.497711 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.130:0
--local_ip_for_outbound_sockets=127.5.225.130
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:40433
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.130.keytab
--principal=oryx/127.5.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:45741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:36.816848 13812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:36.817181 13812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:36.817343 13812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:36.824708 13812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:36.824821 13812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:36.824905 13812 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:42:36.824985 13812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.130
I20260620 18:42:36.834749 13812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:45741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.130.keytab
--principal=oryx/127.5.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.225.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.225.190:40433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.130
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:36.836032 13812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:36.837548 13812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:36.851270 13817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:36.851298 13818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:36.852150 13812 server_base.cc:1034] running on GCE node
W20260620 18:42:36.852275 13820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:36.853026 13812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:36.854085 13812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:36.855305 13812 hybrid_clock.cc:648] HybridClock initialized: now 1781980956855236 us; error 39 us; skew 500 ppm
I20260620 18:42:36.855655 13812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:36 dist-test-slave-k6lg krb5kdc[13556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980956, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:36.859045 13812 init.cc:377] Logged in from keytab as oryx/127.5.225.130@KRBTEST.COM (short username oryx)
I20260620 18:42:36.860579 13812 webserver.cc:460] Webserver started at http://127.5.225.130:32775/ using document root <none> and password file <none>
I20260620 18:42:36.861316 13812 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:36.861426 13812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:36.861756 13812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:36.864770 13812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-1/data/instance:
uuid: "3cf75338fddf4a90b0628d178afd67f1"
format_stamp: "Formatted at 2026-06-20 18:42:36 on dist-test-slave-k6lg"
server_key: "d91b36af573c8523d854a936358c1c3d"
server_key_iv: "2412a6a4e0b80f0c15240792ef74583a"
server_key_version: "encryptionkey@0"
I20260620 18:42:36.865556 13812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance:
uuid: "3cf75338fddf4a90b0628d178afd67f1"
format_stamp: "Formatted at 2026-06-20 18:42:36 on dist-test-slave-k6lg"
server_key: "d91b36af573c8523d854a936358c1c3d"
server_key_iv: "2412a6a4e0b80f0c15240792ef74583a"
server_key_version: "encryptionkey@0"
I20260620 18:42:36.870755 13812 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 18:42:36.874413 13827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:36.875938 13812 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:42:36.876214 13812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-1/wal
uuid: "3cf75338fddf4a90b0628d178afd67f1"
format_stamp: "Formatted at 2026-06-20 18:42:36 on dist-test-slave-k6lg"
server_key: "d91b36af573c8523d854a936358c1c3d"
server_key_iv: "2412a6a4e0b80f0c15240792ef74583a"
server_key_version: "encryptionkey@0"
I20260620 18:42:36.876439 13812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:36.909691 13812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:36.914057 13812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:36.914367 13812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:36.917145 13812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:36.917250 13812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:36.917380 13812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:36.917447 13812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:42:36 dist-test-slave-k6lg krb5kdc[13556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980956, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.225.130@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM
I20260620 18:42:36.941998 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:36.917866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:58667 (local address 127.5.225.190:40433)
0620 18:42:36.918070 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:36.918078 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:36.919627 (+ 1549us) server_negotiation.cc:407] Connection header received
0620 18:42:36.921840 (+ 2213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:36.921846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:36.921939 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:36.922044 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:36.924253 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:36.925567 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:36.928148 (+ 2581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:36.928955 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:36.929117 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:36.933960 (+ 4843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:36.933971 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:36.933980 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:36.934036 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:36.938395 (+ 4359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:36.939597 (+ 1202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:36.939603 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:36.939610 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:36.939747 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:36.940236 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:36.940239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:36.940242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:36.940539 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:36.940704 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:36.941421 (+ 717us) server_negotiation.cc:299] Negotiation successful
0620 18:42:36.941656 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:42:36.943323 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:36.918575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:40433 (local address 127.5.225.130:58667)
0620 18:42:36.919441 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:36.919482 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:36.921534 (+ 2052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:36.922359 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:36.922375 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:36.923192 (+ 817us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:36.924022 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:36.924070 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:36.925765 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:36.925775 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:36.927958 (+ 2183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:36.927971 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:36.929222 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:36.929228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:36.929419 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:36.930284 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:36.930307 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:36.933618 (+ 3311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:36.938600 (+ 4982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:36.938608 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:36.938623 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:36.939455 (+ 832us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:36.939892 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:36.939896 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:36.939898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:36.940104 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:36.940672 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:36.940680 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:36.941049 (+ 369us) client_negotiation.cc:769] Sending connection context
0620 18:42:36.941304 (+ 255us) client_negotiation.cc:240] Negotiation successful
0620 18:42:36.941864 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":383,"threads_started":1}
I20260620 18:42:36.966099 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:36.950287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:40433 (local address 127.5.225.130:41759)
0620 18:42:36.951046 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:36.951063 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:36.951187 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:36.951551 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:36.951554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:36.952017 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:36.952542 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:36.952560 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:36.953722 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:36.953727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:36.954812 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:36.954823 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:36.955731 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:36.955740 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:36.956013 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:36.956802 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:36.956833 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:36.958135 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:36.962716 (+ 4581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:36.962724 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:36.962729 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:36.963301 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:36.963796 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:36.963800 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:36.963802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:36.963934 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:36.964738 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:36.964746 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:36.965021 (+ 275us) client_negotiation.cc:769] Sending connection context
0620 18:42:36.965208 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 18:42:36.965631 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":456,"threads_started":1}
I20260620 18:42:36.966245 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:36.950187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:41759 (local address 127.5.225.190:40433)
0620 18:42:36.950382 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:36.950389 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:36.951145 (+ 756us) server_negotiation.cc:407] Connection header received
0620 18:42:36.951302 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:36.951311 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:36.951403 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:36.951531 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:36.952689 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:36.953577 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:36.954958 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:36.955515 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:36.955662 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:36.958430 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:36.958436 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:36.958441 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:36.958481 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:36.962514 (+ 4033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:36.963485 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:36.963493 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:36.963499 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:36.963669 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:36.964059 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:36.964064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:36.964066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:36.964560 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:36.965511 (+ 951us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:36.965669 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 18:42:36.965886 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:42:36.973102 13812 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.130:39239
I20260620 18:42:36.973127 13946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.130:39239 every 8 connection(s)
I20260620 18:42:36.974391 13812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
I20260620 18:42:36.976394 13947 heartbeater.cc:347] Connected to a master server at 127.5.225.190:40433
I20260620 18:42:36.976612 13947 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:36.977226 13947 heartbeater.cc:511] Master 127.5.225.190:40433 requested a full tablet report, sending...
I20260620 18:42:36.978574 13601 ts_manager.cc:194] Registered new tserver with Master: 3cf75338fddf4a90b0628d178afd67f1 (127.5.225.130:39239)
I20260620 18:42:36.978888 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 13812
I20260620 18:42:36.979050 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance
I20260620 18:42:36.979476 6022 external_mini_cluster.cc:1428] Setting key f3311c857d16af09f27e831c1fa63617
I20260620 18:42:36.979593 13601 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.225.130@KRBTEST.COM'} at 127.5.225.130:58667
WARNING: no policy specified for oryx/127.5.225.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.225.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.131.keytab.
Entry for principal oryx/127.5.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.131.keytab.
I20260620 18:42:37.025059 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.131:0
--local_ip_for_outbound_sockets=127.5.225.131
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:40433
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.131.keytab
--principal=oryx/127.5.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:45741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:37.348184 13951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:37.348492 13951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:37.348649 13951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:37.355841 13951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:37.355985 13951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:37.356120 13951 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:42:37.356248 13951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.131
I20260620 18:42:37.365893 13951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:45741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.131.keytab
--principal=oryx/127.5.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.225.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.225.190:40433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.131
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:37.367215 13951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:37.368659 13951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:37.382098 13956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:37.382134 13957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:37.382061 13959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:37.383484 13951 server_base.cc:1034] running on GCE node
I20260620 18:42:37.384084 13951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:37.384971 13951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:37.386241 13951 hybrid_clock.cc:648] HybridClock initialized: now 1781980957386136 us; error 44 us; skew 500 ppm
I20260620 18:42:37.386525 13951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:37 dist-test-slave-k6lg krb5kdc[13556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980957, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:37.389536 13951 init.cc:377] Logged in from keytab as oryx/127.5.225.131@KRBTEST.COM (short username oryx)
I20260620 18:42:37.391000 13951 webserver.cc:460] Webserver started at http://127.5.225.131:33003/ using document root <none> and password file <none>
I20260620 18:42:37.391702 13951 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:37.391798 13951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:37.392063 13951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:37.395165 13951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-2/data/instance:
uuid: "789b964919a043e6a2b7130b340139af"
format_stamp: "Formatted at 2026-06-20 18:42:37 on dist-test-slave-k6lg"
server_key: "3ec6ac54f43164aa3809c2a65256d620"
server_key_iv: "f56c9e3f1a07a0c46cd76678f6f21da5"
server_key_version: "encryptionkey@0"
I20260620 18:42:37.396235 13951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance:
uuid: "789b964919a043e6a2b7130b340139af"
format_stamp: "Formatted at 2026-06-20 18:42:37 on dist-test-slave-k6lg"
server_key: "3ec6ac54f43164aa3809c2a65256d620"
server_key_iv: "f56c9e3f1a07a0c46cd76678f6f21da5"
server_key_version: "encryptionkey@0"
I20260620 18:42:37.402990 13951 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260620 18:42:37.406320 13966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:37.407565 13951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:42:37.407735 13951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-2/wal
uuid: "789b964919a043e6a2b7130b340139af"
format_stamp: "Formatted at 2026-06-20 18:42:37 on dist-test-slave-k6lg"
server_key: "3ec6ac54f43164aa3809c2a65256d620"
server_key_iv: "f56c9e3f1a07a0c46cd76678f6f21da5"
server_key_version: "encryptionkey@0"
I20260620 18:42:37.407941 13951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:37.433135 13951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:37.437485 13951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:37.437768 13951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:37.440478 13951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:37.440582 13951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:37.440737 13951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:37.440819 13951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:42:37 dist-test-slave-k6lg krb5kdc[13556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980957, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.225.131@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM
I20260620 18:42:37.457581 13808 heartbeater.cc:503] Master 127.5.225.190:40433 was elected leader, sending a full tablet report...
I20260620 18:42:37.463146 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:37.440983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:53447 (local address 127.5.225.190:40433)
0620 18:42:37.441223 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:37.441231 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:37.442700 (+ 1469us) server_negotiation.cc:407] Connection header received
0620 18:42:37.445081 (+ 2381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:37.445087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:37.445183 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:37.445305 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:37.447261 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:37.448519 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:37.450660 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:37.451393 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:37.451524 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:37.455798 (+ 4274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:37.455809 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:37.455818 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:37.455876 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:37.459536 (+ 3660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:37.460563 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:37.460572 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:37.460579 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:37.460763 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:37.461188 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:37.461196 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:37.461202 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:37.461674 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:37.461900 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:37.462568 (+ 668us) server_negotiation.cc:299] Negotiation successful
0620 18:42:37.462791 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260620 18:42:37.464125 13975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:37.441663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:40433 (local address 127.5.225.131:53447)
0620 18:42:37.442548 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:37.442586 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:37.444740 (+ 2154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:37.445526 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:37.445534 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:37.446394 (+ 860us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:37.447084 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:37.447102 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:37.448743 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:37.448750 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:37.450443 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:37.450465 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:37.451636 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:37.451643 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:37.451841 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:37.452559 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:37.452599 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:37.455514 (+ 2915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:37.459739 (+ 4225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:37.459747 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:37.459762 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:37.460394 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:37.460885 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:37.460888 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:37.460890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:37.461074 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:37.461798 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:37.461804 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:37.462136 (+ 332us) client_negotiation.cc:769] Sending connection context
0620 18:42:37.462467 (+ 331us) client_negotiation.cc:240] Negotiation successful
0620 18:42:37.462998 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":394,"threads_started":1}
I20260620 18:42:37.488658 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:37.472238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:46241 (local address 127.5.225.190:40433)
0620 18:42:37.472456 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:37.472464 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:37.473194 (+ 730us) server_negotiation.cc:407] Connection header received
0620 18:42:37.473410 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:37.473414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:37.473473 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:37.473547 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:37.475122 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:37.476015 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:37.478704 (+ 2689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:37.479297 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:37.479443 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:37.482555 (+ 3112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:37.482560 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:37.482563 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:37.482612 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:37.485443 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:37.486396 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:37.486400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:37.486403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:37.486556 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:37.487059 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:37.487065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:37.487071 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:37.487498 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:37.487655 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:37.488052 (+ 397us) server_negotiation.cc:299] Negotiation successful
0620 18:42:37.488276 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260620 18:42:37.488732 14021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:37.472313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:40433 (local address 127.5.225.131:46241)
0620 18:42:37.473074 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:37.473099 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:37.473280 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:37.473606 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:37.473614 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:37.474386 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:37.474874 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:37.474895 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:37.476263 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:37.476273 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:37.478462 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:37.478483 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:37.479568 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:37.479577 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:37.479874 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:37.480738 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:37.480768 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:37.482321 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:37.485580 (+ 3259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:37.485585 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:37.485589 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:37.486226 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:37.486690 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:37.486695 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:37.486702 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:37.486883 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:37.487632 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:37.487636 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:37.487792 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 18:42:37.487933 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 18:42:37.488276 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":354,"threads_started":1}
I20260620 18:42:37.503142 13951 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.131:37959
I20260620 18:42:37.503155 14085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.131:37959 every 8 connection(s)
I20260620 18:42:37.504360 13951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
I20260620 18:42:37.505826 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 13951
I20260620 18:42:37.505998 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance
I20260620 18:42:37.506573 6022 external_mini_cluster.cc:1428] Setting key 14ec867ede1b4e801223e88c787cfc0a
I20260620 18:42:37.506559 14086 heartbeater.cc:347] Connected to a master server at 127.5.225.190:40433
I20260620 18:42:37.507413 14086 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:37.508190 14086 heartbeater.cc:511] Master 127.5.225.190:40433 requested a full tablet report, sending...
I20260620 18:42:37.509470 13601 ts_manager.cc:194] Registered new tserver with Master: 789b964919a043e6a2b7130b340139af (127.5.225.131:37959)
I20260620 18:42:37.510358 13601 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.225.131@KRBTEST.COM'} at 127.5.225.131:53447
I20260620 18:42:37.521764 6022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 18:42:37 dist-test-slave-k6lg krb5kdc[13556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980957, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 18:42:37 dist-test-slave-k6lg krb5kdc[13556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:42:37 dist-test-slave-k6lg krb5kdc[13556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:42:37.982437 13947 heartbeater.cc:503] Master 127.5.225.190:40433 was elected leader, sending a full tablet report...
I20260620 18:42:38.512799 14086 heartbeater.cc:503] Master 127.5.225.190:40433 was elected leader, sending a full tablet report...
W20260620 18:42:38.630074 14094 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.066s user 0.001s sys 0.013s
W20260620 18:42:38.631286 13696 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:37.559193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38536 (local address 127.5.225.190:40433)
0620 18:42:37.559358 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:37.559363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:37.559618 (+ 255us) server_negotiation.cc:407] Connection header received
0620 18:42:37.559780 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:37.559785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:37.559869 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:37.559967 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:37.561223 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:37.561949 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:37.563187 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:37.563633 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:37.563734 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:38.631052 (+1067318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38536: BlockingRecv error: recv got EOF from 127.0.0.1:38536 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 18:42:38 dist-test-slave-k6lg krb5kdc[13556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980957, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM
I20260620 18:42:38.652408 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:38.635784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38540 (local address 127.5.225.190:40433)
0620 18:42:38.636014 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:38.636021 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:38.636309 (+ 288us) server_negotiation.cc:407] Connection header received
0620 18:42:38.636442 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:38.636446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:38.636534 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:38.636624 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:38.637827 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:38.638603 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:38.639764 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:38.640148 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:38.640246 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:38.647321 (+ 7075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:38.647326 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:38.647329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:38.647365 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:38.649789 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:38.650527 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:38.650534 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:38.650537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:38.650660 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:38.651017 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:38.651023 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:38.651026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:38.651310 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:38.651579 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:38.651859 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 18:42:38.652085 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:42:38.656231 13601 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38540:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:42:38.662899 13601 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:42:38.698870 14106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:38.674632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57452 (local address 127.5.225.131:37959)
0620 18:42:38.689710 (+ 15078us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:38.689721 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:38.689758 (+ 37us) server_negotiation.cc:407] Connection header received
0620 18:42:38.689880 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:38.689891 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:38.690221 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:38.690453 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:38.692268 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:38.693835 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:38.696442 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:38.697722 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:38.697862 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:38.698073 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:38.698379 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 18:42:38.698559 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14910,"thread_start_us":14774,"threads_started":1}
I20260620 18:42:38.698930 14108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:38.674372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.131:37959 (local address 127.0.0.1:57452)
0620 18:42:38.687812 (+ 13440us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:38.687862 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:38.688077 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:38.690753 (+ 2676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:38.690765 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:38.690814 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:38.691858 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:38.691871 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:38.693987 (+ 2116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:38.693993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:38.696252 (+ 2259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:38.696263 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:38.697928 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:38.697934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:38.698113 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:38.698183 (+ 70us) client_negotiation.cc:769] Sending connection context
0620 18:42:38.698320 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 18:42:38.698506 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13087,"mutex_wait_us":4,"thread_start_us":15812,"threads_started":1}
I20260620 18:42:38.700767 14110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:38.675726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57858 (local address 127.5.225.129:44023)
0620 18:42:38.683516 (+ 7790us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:38.683529 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:38.693302 (+ 9773us) server_negotiation.cc:407] Connection header received
0620 18:42:38.693482 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:38.693488 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:38.693636 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:38.693823 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:38.694436 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:38.695992 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:38.697930 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:38.699399 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:38.699504 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:38.699703 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:38.700336 (+ 633us) server_negotiation.cc:299] Negotiation successful
0620 18:42:38.700501 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7572,"thread_start_us":7394,"threads_started":1}
I20260620 18:42:38.700724 14107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:38.675183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:44023 (local address 127.0.0.1:57858)
0620 18:42:38.693175 (+ 17992us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:38.693189 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:38.693324 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:38.693808 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:38.693812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:38.693835 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:38.694256 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:38.694274 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:38.696158 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:38.696169 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:38.697738 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:38.697750 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:38.699962 (+ 2212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:38.699971 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:38.700133 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:38.700173 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 18:42:38.700264 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 18:42:38.700375 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17867,"mutex_wait_us":22,"thread_start_us":13122,"threads_started":1}
I20260620 18:42:38.702442 14109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:38.675359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46782 (local address 127.5.225.130:39239)
0620 18:42:38.690747 (+ 15388us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:38.690752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:38.690774 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:42:38.690842 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:38.690847 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:38.690990 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:38.691175 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:38.692936 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:38.694389 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:38.696917 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:38.698379 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:38.698510 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:38.698693 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:38.699429 (+ 736us) server_negotiation.cc:299] Negotiation successful
0620 18:42:38.699603 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15273,"thread_start_us":15064,"threads_started":1}
I20260620 18:42:38.702898 14105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:38.675029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.130:39239 (local address 127.0.0.1:46782)
0620 18:42:38.690031 (+ 15002us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:38.690065 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:38.690412 (+ 347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:38.691166 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:38.691171 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:38.691200 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:38.692737 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:38.692753 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:38.694507 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:38.694512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:38.696754 (+ 2242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:38.696767 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:38.698838 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:38.698844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:38.699039 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:38.699091 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 18:42:38.699173 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 18:42:38.700075 (+ 902us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14821,"mutex_wait_us":28,"thread_start_us":17882,"threads_started":1}
I20260620 18:42:38.702975 14020 tablet_service.cc:1467] Processing CreateTablet for tablet 6675b514692e483c9198399b47dc7033 (DEFAULT_TABLE table=test-table [id=60156301db2c4482b821adbdbf71b39d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:42:38.703833 13738 tablet_service.cc:1467] Processing CreateTablet for tablet 6675b514692e483c9198399b47dc7033 (DEFAULT_TABLE table=test-table [id=60156301db2c4482b821adbdbf71b39d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:42:38.704306 14020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6675b514692e483c9198399b47dc7033. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:38.704994 13738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6675b514692e483c9198399b47dc7033. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:38.713620 13877 tablet_service.cc:1467] Processing CreateTablet for tablet 6675b514692e483c9198399b47dc7033 (DEFAULT_TABLE table=test-table [id=60156301db2c4482b821adbdbf71b39d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:42:38.715000 13877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6675b514692e483c9198399b47dc7033. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:38.715337 14111 tablet_bootstrap.cc:492] T 6675b514692e483c9198399b47dc7033 P 06416c21938742de87f05ca67fcc42dc: Bootstrap starting.
I20260620 18:42:38.718616 14111 tablet_bootstrap.cc:654] T 6675b514692e483c9198399b47dc7033 P 06416c21938742de87f05ca67fcc42dc: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:38.719571 14111 log.cc:824] T 6675b514692e483c9198399b47dc7033 P 06416c21938742de87f05ca67fcc42dc: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:38.723088 14112 tablet_bootstrap.cc:492] T 6675b514692e483c9198399b47dc7033 P 789b964919a043e6a2b7130b340139af: Bootstrap starting.
I20260620 18:42:38.725009 14111 tablet_bootstrap.cc:492] T 6675b514692e483c9198399b47dc7033 P 06416c21938742de87f05ca67fcc42dc: No bootstrap required, opened a new log
I20260620 18:42:38.725270 14111 ts_tablet_manager.cc:1397] T 6675b514692e483c9198399b47dc7033 P 06416c21938742de87f05ca67fcc42dc: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 18:42:38.726545 14112 tablet_bootstrap.cc:654] T 6675b514692e483c9198399b47dc7033 P 789b964919a043e6a2b7130b340139af: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:38.727854 14112 log.cc:824] T 6675b514692e483c9198399b47dc7033 P 789b964919a043e6a2b7130b340139af: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:38.729745 14115 tablet_bootstrap.cc:492] T 6675b514692e483c9198399b47dc7033 P 3cf75338fddf4a90b0628d178afd67f1: Bootstrap starting.
I20260620 18:42:38.730791 14112 tablet_bootstrap.cc:492] T 6675b514692e483c9198399b47dc7033 P 789b964919a043e6a2b7130b340139af: No bootstrap required, opened a new log
I20260620 18:42:38.731045 14112 ts_tablet_manager.cc:1397] T 6675b514692e483c9198399b47dc7033 P 789b964919a043e6a2b7130b340139af: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 18:42:38.732890 14115 tablet_bootstrap.cc:654] T 6675b514692e483c9198399b47dc7033 P 3cf75338fddf4a90b0628d178afd67f1: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:38.733901 14115 log.cc:824] T 6675b514692e483c9198399b47dc7033 P 3cf75338fddf4a90b0628d178afd67f1: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:38.735345 14111 raft_consensus.cc:348] T 6675b514692e483c9198399b47dc7033 P 06416c21938742de87f05ca67fcc42dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } }
I20260620 18:42:38.735781 14111 raft_consensus.cc:374] T 6675b514692e483c9198399b47dc7033 P 06416c21938742de87f05ca67fcc42dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:38.735879 14111 raft_consensus.cc:729] T 6675b514692e483c9198399b47dc7033 P 06416c21938742de87f05ca67fcc42dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06416c21938742de87f05ca67fcc42dc, State: Initialized, Role: FOLLOWER
I20260620 18:42:38.736485 14115 tablet_bootstrap.cc:492] T 6675b514692e483c9198399b47dc7033 P 3cf75338fddf4a90b0628d178afd67f1: No bootstrap required, opened a new log
I20260620 18:42:38.736477 14111 consensus_queue.cc:260] T 6675b514692e483c9198399b47dc7033 P 06416c21938742de87f05ca67fcc42dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } }
I20260620 18:42:38.736799 14115 ts_tablet_manager.cc:1397] T 6675b514692e483c9198399b47dc7033 P 3cf75338fddf4a90b0628d178afd67f1: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260620 18:42:38.742774 14112 raft_consensus.cc:348] T 6675b514692e483c9198399b47dc7033 P 789b964919a043e6a2b7130b340139af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } }
I20260620 18:42:38.743259 14112 raft_consensus.cc:374] T 6675b514692e483c9198399b47dc7033 P 789b964919a043e6a2b7130b340139af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:38.743405 14112 raft_consensus.cc:729] T 6675b514692e483c9198399b47dc7033 P 789b964919a043e6a2b7130b340139af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789b964919a043e6a2b7130b340139af, State: Initialized, Role: FOLLOWER
I20260620 18:42:38.743552 14111 ts_tablet_manager.cc:1428] T 6675b514692e483c9198399b47dc7033 P 06416c21938742de87f05ca67fcc42dc: Time spent starting tablet: real 0.018s user 0.015s sys 0.004s
I20260620 18:42:38.744177 14112 consensus_queue.cc:260] T 6675b514692e483c9198399b47dc7033 P 789b964919a043e6a2b7130b340139af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } }
I20260620 18:42:38.744180 14115 raft_consensus.cc:348] T 6675b514692e483c9198399b47dc7033 P 3cf75338fddf4a90b0628d178afd67f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } }
I20260620 18:42:38.744590 14115 raft_consensus.cc:374] T 6675b514692e483c9198399b47dc7033 P 3cf75338fddf4a90b0628d178afd67f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:38.744729 14115 raft_consensus.cc:729] T 6675b514692e483c9198399b47dc7033 P 3cf75338fddf4a90b0628d178afd67f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cf75338fddf4a90b0628d178afd67f1, State: Initialized, Role: FOLLOWER
I20260620 18:42:38.745343 14115 consensus_queue.cc:260] T 6675b514692e483c9198399b47dc7033 P 3cf75338fddf4a90b0628d178afd67f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } }
I20260620 18:42:38.745834 14112 ts_tablet_manager.cc:1428] T 6675b514692e483c9198399b47dc7033 P 789b964919a043e6a2b7130b340139af: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260620 18:42:38.747496 14115 ts_tablet_manager.cc:1428] T 6675b514692e483c9198399b47dc7033 P 3cf75338fddf4a90b0628d178afd67f1: Time spent starting tablet: real 0.011s user 0.009s sys 0.001s
W20260620 18:42:38.756769 14087 tablet.cc:2369] T 6675b514692e483c9198399b47dc7033 P 789b964919a043e6a2b7130b340139af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:42:38.811398 14118 raft_consensus.cc:482] T 6675b514692e483c9198399b47dc7033 P 789b964919a043e6a2b7130b340139af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:42:38.811650 14118 raft_consensus.cc:504] T 6675b514692e483c9198399b47dc7033 P 789b964919a043e6a2b7130b340139af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } }
I20260620 18:42:38.813256 14118 leader_election.cc:288] T 6675b514692e483c9198399b47dc7033 P 789b964919a043e6a2b7130b340139af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cf75338fddf4a90b0628d178afd67f1 (127.5.225.130:39239), 06416c21938742de87f05ca67fcc42dc (127.5.225.129:44023)
I20260620 18:42:38.820022 14109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:38.813657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:57619 (local address 127.5.225.130:39239)
0620 18:42:38.813890 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:38.813900 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:38.814702 (+ 802us) server_negotiation.cc:407] Connection header received
0620 18:42:38.814898 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:38.814901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:38.814959 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:38.815053 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:38.815663 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:38.816348 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:38.818304 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:38.819151 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:38.819327 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:38.819401 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:38.819654 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 18:42:38.819774 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:42:38.820173 14110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:38.813794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:42331 (local address 127.5.225.129:44023)
0620 18:42:38.814035 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:38.814044 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:38.814616 (+ 572us) server_negotiation.cc:407] Connection header received
0620 18:42:38.814841 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:38.814846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:38.814904 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:38.814998 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:38.815565 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:38.816283 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:38.818406 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:38.819258 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:38.819452 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:38.819525 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:38.819803 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 18:42:38.819927 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:42:38.820211 14121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:38.813675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.130:39239 (local address 127.5.225.131:57619)
0620 18:42:38.814593 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:38.814607 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:38.814751 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:38.815064 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:38.815069 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:38.815095 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:38.815512 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:38.815520 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:38.816481 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:38.816486 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:38.818102 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:38.818115 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:38.819343 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:38.819349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:38.819504 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:38.819527 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 18:42:38.819631 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 18:42:38.819750 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"mutex_wait_us":14,"thread_start_us":618,"threads_started":1}
I20260620 18:42:38.820317 14120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:38.813597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:44023 (local address 127.5.225.131:42331)
0620 18:42:38.814493 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:38.814510 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:38.814686 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:38.815013 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:38.815017 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:38.815041 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:38.815412 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:38.815425 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:38.816411 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:38.816419 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:38.818257 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:38.818268 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:38.819491 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:38.819498 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:38.819648 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:38.819672 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 18:42:38.819777 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 18:42:38.819898 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"mutex_wait_us":78,"thread_start_us":520,"threads_started":1}
I20260620 18:42:38.821313 13763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6675b514692e483c9198399b47dc7033" candidate_uuid: "789b964919a043e6a2b7130b340139af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06416c21938742de87f05ca67fcc42dc" is_pre_election: true
I20260620 18:42:38.821329 13902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6675b514692e483c9198399b47dc7033" candidate_uuid: "789b964919a043e6a2b7130b340139af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cf75338fddf4a90b0628d178afd67f1" is_pre_election: true
I20260620 18:42:38.821767 13902 raft_consensus.cc:2454] T 6675b514692e483c9198399b47dc7033 P 3cf75338fddf4a90b0628d178afd67f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 789b964919a043e6a2b7130b340139af in term 0.
I20260620 18:42:38.821776 13763 raft_consensus.cc:2454] T 6675b514692e483c9198399b47dc7033 P 06416c21938742de87f05ca67fcc42dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 789b964919a043e6a2b7130b340139af in term 0.
I20260620 18:42:38.822362 13970 leader_election.cc:302] T 6675b514692e483c9198399b47dc7033 P 789b964919a043e6a2b7130b340139af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06416c21938742de87f05ca67fcc42dc, 789b964919a043e6a2b7130b340139af; no voters:
I20260620 18:42:38.822759 14118 raft_consensus.cc:2781] T 6675b514692e483c9198399b47dc7033 P 789b964919a043e6a2b7130b340139af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:42:38.822903 14118 raft_consensus.cc:482] T 6675b514692e483c9198399b47dc7033 P 789b964919a043e6a2b7130b340139af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:42:38.823025 14118 raft_consensus.cc:3037] T 6675b514692e483c9198399b47dc7033 P 789b964919a043e6a2b7130b340139af [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:38.825124 14118 raft_consensus.cc:504] T 6675b514692e483c9198399b47dc7033 P 789b964919a043e6a2b7130b340139af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } }
I20260620 18:42:38.825802 14118 leader_election.cc:288] T 6675b514692e483c9198399b47dc7033 P 789b964919a043e6a2b7130b340139af [CANDIDATE]: Term 1 election: Requested vote from peers 3cf75338fddf4a90b0628d178afd67f1 (127.5.225.130:39239), 06416c21938742de87f05ca67fcc42dc (127.5.225.129:44023)
I20260620 18:42:38.826238 13902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6675b514692e483c9198399b47dc7033" candidate_uuid: "789b964919a043e6a2b7130b340139af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cf75338fddf4a90b0628d178afd67f1"
I20260620 18:42:38.826429 13902 raft_consensus.cc:3037] T 6675b514692e483c9198399b47dc7033 P 3cf75338fddf4a90b0628d178afd67f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:38.826421 13763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6675b514692e483c9198399b47dc7033" candidate_uuid: "789b964919a043e6a2b7130b340139af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06416c21938742de87f05ca67fcc42dc"
I20260620 18:42:38.826700 13763 raft_consensus.cc:3037] T 6675b514692e483c9198399b47dc7033 P 06416c21938742de87f05ca67fcc42dc [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:38.828553 13902 raft_consensus.cc:2454] T 6675b514692e483c9198399b47dc7033 P 3cf75338fddf4a90b0628d178afd67f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 789b964919a043e6a2b7130b340139af in term 1.
I20260620 18:42:38.828949 13967 leader_election.cc:302] T 6675b514692e483c9198399b47dc7033 P 789b964919a043e6a2b7130b340139af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cf75338fddf4a90b0628d178afd67f1, 789b964919a043e6a2b7130b340139af; no voters:
I20260620 18:42:38.829133 13763 raft_consensus.cc:2454] T 6675b514692e483c9198399b47dc7033 P 06416c21938742de87f05ca67fcc42dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 789b964919a043e6a2b7130b340139af in term 1.
I20260620 18:42:38.829242 14118 raft_consensus.cc:2781] T 6675b514692e483c9198399b47dc7033 P 789b964919a043e6a2b7130b340139af [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:42:38.829793 14118 raft_consensus.cc:686] T 6675b514692e483c9198399b47dc7033 P 789b964919a043e6a2b7130b340139af [term 1 LEADER]: Becoming Leader. State: Replica: 789b964919a043e6a2b7130b340139af, State: Running, Role: LEADER
I20260620 18:42:38.830330 14118 consensus_queue.cc:237] T 6675b514692e483c9198399b47dc7033 P 789b964919a043e6a2b7130b340139af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } }
I20260620 18:42:38.835136 13600 catalog_manager.cc:5495] T 6675b514692e483c9198399b47dc7033 P 789b964919a043e6a2b7130b340139af reported cstate change: term changed from 0 to 1, leader changed from <none> to 789b964919a043e6a2b7130b340139af (127.5.225.131). New cstate: current_term: 1 leader_uuid: "789b964919a043e6a2b7130b340139af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } health_report { overall_health: UNKNOWN } } }
I20260620 18:42:38.852596 14106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:38.845993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57464 (local address 127.5.225.131:37959)
0620 18:42:38.846220 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:38.846245 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:38.846270 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:42:38.846477 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:38.846483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:38.846592 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:38.846752 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:42:38.847066 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:38.848274 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:38.849817 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:38.850542 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:38.850678 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:38.851163 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:42:38.851895 (+ 732us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:42:38.852057 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:38.852203 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 18:42:38.852300 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:42:38.861299 13902 raft_consensus.cc:1261] T 6675b514692e483c9198399b47dc7033 P 3cf75338fddf4a90b0628d178afd67f1 [term 1 FOLLOWER]: Refusing update from remote peer 789b964919a043e6a2b7130b340139af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:42:38.861294 13763 raft_consensus.cc:1261] T 6675b514692e483c9198399b47dc7033 P 06416c21938742de87f05ca67fcc42dc [term 1 FOLLOWER]: Refusing update from remote peer 789b964919a043e6a2b7130b340139af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:42:38.862419 14122 consensus_queue.cc:1035] T 6675b514692e483c9198399b47dc7033 P 789b964919a043e6a2b7130b340139af [LEADER]: Connected to new peer: Peer: permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:42:38.862797 14118 consensus_queue.cc:1035] T 6675b514692e483c9198399b47dc7033 P 789b964919a043e6a2b7130b340139af [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:42:38.875471 14126 mvcc.cc:205] Tried to move back new op lower bound from 7298994007486193664 to 7298994007372697600. Current Snapshot: MvccSnapshot[applied={T|T < 7298994007486193664}]
I20260620 18:42:38.875803 14125 mvcc.cc:205] Tried to move back new op lower bound from 7298994007486193664 to 7298994007372697600. Current Snapshot: MvccSnapshot[applied={T|T < 7298994007486193664}]
I20260620 18:42:38.893311 13599 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:42:38.893791 13599 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:42:38.897946 13599 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 972f20065ea9422a838127c325ed3908: Sending DeleteTablet for 3 replicas of tablet 6675b514692e483c9198399b47dc7033
I20260620 18:42:38.898962 14020 tablet_service.cc:1514] Processing DeleteTablet for tablet 6675b514692e483c9198399b47dc7033 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:42:38 UTC) from {username='oryx', principal='oryx/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:57452
I20260620 18:42:38.899158 13877 tablet_service.cc:1514] Processing DeleteTablet for tablet 6675b514692e483c9198399b47dc7033 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:42:38 UTC) from {username='oryx', principal='oryx/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:46782
I20260620 18:42:38.899461 13738 tablet_service.cc:1514] Processing DeleteTablet for tablet 6675b514692e483c9198399b47dc7033 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:42:38 UTC) from {username='oryx', principal='oryx/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:57858
I20260620 18:42:38.900012 14135 tablet_replica.cc:330] T 6675b514692e483c9198399b47dc7033 P 789b964919a043e6a2b7130b340139af: stopping tablet replica
I20260620 18:42:38.900120 14136 tablet_replica.cc:330] T 6675b514692e483c9198399b47dc7033 P 3cf75338fddf4a90b0628d178afd67f1: stopping tablet replica
I20260620 18:42:38.900548 14135 raft_consensus.cc:2229] T 6675b514692e483c9198399b47dc7033 P 789b964919a043e6a2b7130b340139af [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:42:38.900574 14136 raft_consensus.cc:2229] T 6675b514692e483c9198399b47dc7033 P 3cf75338fddf4a90b0628d178afd67f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:42:38.901232 14136 raft_consensus.cc:2258] T 6675b514692e483c9198399b47dc7033 P 3cf75338fddf4a90b0628d178afd67f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:42:38.901237 14135 raft_consensus.cc:2258] T 6675b514692e483c9198399b47dc7033 P 789b964919a043e6a2b7130b340139af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:42:38.901325 13599 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38540:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:42:38.902913 13599 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:42:38.903403 14135 ts_tablet_manager.cc:1905] T 6675b514692e483c9198399b47dc7033 P 789b964919a043e6a2b7130b340139af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:42:38.903487 14136 ts_tablet_manager.cc:1905] T 6675b514692e483c9198399b47dc7033 P 3cf75338fddf4a90b0628d178afd67f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:42:38.906605 14137 tablet_replica.cc:330] T 6675b514692e483c9198399b47dc7033 P 06416c21938742de87f05ca67fcc42dc: stopping tablet replica
I20260620 18:42:38.907045 14137 raft_consensus.cc:2229] T 6675b514692e483c9198399b47dc7033 P 06416c21938742de87f05ca67fcc42dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:42:38.907559 14137 raft_consensus.cc:2258] T 6675b514692e483c9198399b47dc7033 P 06416c21938742de87f05ca67fcc42dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:42:38.911584 14135 ts_tablet_manager.cc:1918] T 6675b514692e483c9198399b47dc7033 P 789b964919a043e6a2b7130b340139af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:42:38.911815 14135 log.cc:1196] T 6675b514692e483c9198399b47dc7033 P 789b964919a043e6a2b7130b340139af: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-2/wal/wals/6675b514692e483c9198399b47dc7033
I20260620 18:42:38.912029 13877 tablet_service.cc:1467] Processing CreateTablet for tablet afd148b67e7144d9bfb380e47a1bf33b (DEFAULT_TABLE table=test-table [id=e7bf7528287e47afb06a80bfc98572e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:42:38.912078 14020 tablet_service.cc:1467] Processing CreateTablet for tablet afd148b67e7144d9bfb380e47a1bf33b (DEFAULT_TABLE table=test-table [id=e7bf7528287e47afb06a80bfc98572e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:42:38.912462 13877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afd148b67e7144d9bfb380e47a1bf33b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:38.912485 14135 ts_tablet_manager.cc:1939] T 6675b514692e483c9198399b47dc7033 P 789b964919a043e6a2b7130b340139af: Deleting consensus metadata
I20260620 18:42:38.912492 14020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afd148b67e7144d9bfb380e47a1bf33b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:38.914511 13586 catalog_manager.cc:4841] TS 789b964919a043e6a2b7130b340139af (127.5.225.131:37959): tablet 6675b514692e483c9198399b47dc7033 (table test-table [id=60156301db2c4482b821adbdbf71b39d]) successfully deleted
I20260620 18:42:38.916538 13738 tablet_service.cc:1467] Processing CreateTablet for tablet afd148b67e7144d9bfb380e47a1bf33b (DEFAULT_TABLE table=test-table [id=e7bf7528287e47afb06a80bfc98572e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:42:38.916996 13738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afd148b67e7144d9bfb380e47a1bf33b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:38.918694 14136 ts_tablet_manager.cc:1918] T 6675b514692e483c9198399b47dc7033 P 3cf75338fddf4a90b0628d178afd67f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:42:38.918964 14136 log.cc:1196] T 6675b514692e483c9198399b47dc7033 P 3cf75338fddf4a90b0628d178afd67f1: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-1/wal/wals/6675b514692e483c9198399b47dc7033
I20260620 18:42:38.919570 14136 ts_tablet_manager.cc:1939] T 6675b514692e483c9198399b47dc7033 P 3cf75338fddf4a90b0628d178afd67f1: Deleting consensus metadata
I20260620 18:42:38.919965 14112 tablet_bootstrap.cc:492] T afd148b67e7144d9bfb380e47a1bf33b P 789b964919a043e6a2b7130b340139af: Bootstrap starting.
I20260620 18:42:38.921473 13585 catalog_manager.cc:4841] TS 3cf75338fddf4a90b0628d178afd67f1 (127.5.225.130:39239): tablet 6675b514692e483c9198399b47dc7033 (table test-table [id=60156301db2c4482b821adbdbf71b39d]) successfully deleted
I20260620 18:42:38.921547 14112 tablet_bootstrap.cc:654] T afd148b67e7144d9bfb380e47a1bf33b P 789b964919a043e6a2b7130b340139af: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:38.924997 14115 tablet_bootstrap.cc:492] T afd148b67e7144d9bfb380e47a1bf33b P 3cf75338fddf4a90b0628d178afd67f1: Bootstrap starting.
I20260620 18:42:38.925515 14111 tablet_bootstrap.cc:492] T afd148b67e7144d9bfb380e47a1bf33b P 06416c21938742de87f05ca67fcc42dc: Bootstrap starting.
I20260620 18:42:38.926751 14115 tablet_bootstrap.cc:654] T afd148b67e7144d9bfb380e47a1bf33b P 3cf75338fddf4a90b0628d178afd67f1: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:38.927613 14111 tablet_bootstrap.cc:654] T afd148b67e7144d9bfb380e47a1bf33b P 06416c21938742de87f05ca67fcc42dc: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:38.916991 14137 ts_tablet_manager.cc:1905] T 6675b514692e483c9198399b47dc7033 P 06416c21938742de87f05ca67fcc42dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:42:38.933849 14112 tablet_bootstrap.cc:492] T afd148b67e7144d9bfb380e47a1bf33b P 789b964919a043e6a2b7130b340139af: No bootstrap required, opened a new log
I20260620 18:42:38.934057 14112 ts_tablet_manager.cc:1397] T afd148b67e7144d9bfb380e47a1bf33b P 789b964919a043e6a2b7130b340139af: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.009s
I20260620 18:42:38.934520 14111 tablet_bootstrap.cc:492] T afd148b67e7144d9bfb380e47a1bf33b P 06416c21938742de87f05ca67fcc42dc: No bootstrap required, opened a new log
I20260620 18:42:38.934739 14111 ts_tablet_manager.cc:1397] T afd148b67e7144d9bfb380e47a1bf33b P 06416c21938742de87f05ca67fcc42dc: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260620 18:42:38.935020 14112 raft_consensus.cc:348] T afd148b67e7144d9bfb380e47a1bf33b P 789b964919a043e6a2b7130b340139af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } }
I20260620 18:42:38.935328 14112 raft_consensus.cc:374] T afd148b67e7144d9bfb380e47a1bf33b P 789b964919a043e6a2b7130b340139af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:38.935494 14112 raft_consensus.cc:729] T afd148b67e7144d9bfb380e47a1bf33b P 789b964919a043e6a2b7130b340139af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789b964919a043e6a2b7130b340139af, State: Initialized, Role: FOLLOWER
I20260620 18:42:38.936179 14111 raft_consensus.cc:348] T afd148b67e7144d9bfb380e47a1bf33b P 06416c21938742de87f05ca67fcc42dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } }
I20260620 18:42:38.936419 14111 raft_consensus.cc:374] T afd148b67e7144d9bfb380e47a1bf33b P 06416c21938742de87f05ca67fcc42dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:38.936524 14111 raft_consensus.cc:729] T afd148b67e7144d9bfb380e47a1bf33b P 06416c21938742de87f05ca67fcc42dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06416c21938742de87f05ca67fcc42dc, State: Initialized, Role: FOLLOWER
I20260620 18:42:38.936841 14111 consensus_queue.cc:260] T afd148b67e7144d9bfb380e47a1bf33b P 06416c21938742de87f05ca67fcc42dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } }
I20260620 18:42:38.937090 14115 tablet_bootstrap.cc:492] T afd148b67e7144d9bfb380e47a1bf33b P 3cf75338fddf4a90b0628d178afd67f1: No bootstrap required, opened a new log
I20260620 18:42:38.937251 14115 ts_tablet_manager.cc:1397] T afd148b67e7144d9bfb380e47a1bf33b P 3cf75338fddf4a90b0628d178afd67f1: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.005s
I20260620 18:42:38.937503 14111 ts_tablet_manager.cc:1428] T afd148b67e7144d9bfb380e47a1bf33b P 06416c21938742de87f05ca67fcc42dc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 18:42:38.938072 14115 raft_consensus.cc:348] T afd148b67e7144d9bfb380e47a1bf33b P 3cf75338fddf4a90b0628d178afd67f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } }
I20260620 18:42:38.938335 14115 raft_consensus.cc:374] T afd148b67e7144d9bfb380e47a1bf33b P 3cf75338fddf4a90b0628d178afd67f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:38.938446 14115 raft_consensus.cc:729] T afd148b67e7144d9bfb380e47a1bf33b P 3cf75338fddf4a90b0628d178afd67f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cf75338fddf4a90b0628d178afd67f1, State: Initialized, Role: FOLLOWER
I20260620 18:42:38.936977 14112 consensus_queue.cc:260] T afd148b67e7144d9bfb380e47a1bf33b P 789b964919a043e6a2b7130b340139af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } }
I20260620 18:42:38.938714 14115 consensus_queue.cc:260] T afd148b67e7144d9bfb380e47a1bf33b P 3cf75338fddf4a90b0628d178afd67f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } }
I20260620 18:42:38.939631 14115 ts_tablet_manager.cc:1428] T afd148b67e7144d9bfb380e47a1bf33b P 3cf75338fddf4a90b0628d178afd67f1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:42:38.939679 14112 ts_tablet_manager.cc:1428] T afd148b67e7144d9bfb380e47a1bf33b P 789b964919a043e6a2b7130b340139af: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260620 18:42:38.944604 14137 ts_tablet_manager.cc:1918] T 6675b514692e483c9198399b47dc7033 P 06416c21938742de87f05ca67fcc42dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:42:38.944777 14137 log.cc:1196] T 6675b514692e483c9198399b47dc7033 P 06416c21938742de87f05ca67fcc42dc: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-0/wal/wals/6675b514692e483c9198399b47dc7033
I20260620 18:42:38.945371 14137 ts_tablet_manager.cc:1939] T 6675b514692e483c9198399b47dc7033 P 06416c21938742de87f05ca67fcc42dc: Deleting consensus metadata
I20260620 18:42:38.947270 13588 catalog_manager.cc:4841] TS 06416c21938742de87f05ca67fcc42dc (127.5.225.129:44023): tablet 6675b514692e483c9198399b47dc7033 (table test-table [id=60156301db2c4482b821adbdbf71b39d]) successfully deleted
W20260620 18:42:38.951203 13809 tablet.cc:2369] T afd148b67e7144d9bfb380e47a1bf33b P 06416c21938742de87f05ca67fcc42dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:42:38.977319 13948 tablet.cc:2369] T afd148b67e7144d9bfb380e47a1bf33b P 3cf75338fddf4a90b0628d178afd67f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:42:39.026497 14131 raft_consensus.cc:482] T afd148b67e7144d9bfb380e47a1bf33b P 789b964919a043e6a2b7130b340139af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:42:39.026705 14131 raft_consensus.cc:504] T afd148b67e7144d9bfb380e47a1bf33b P 789b964919a043e6a2b7130b340139af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } }
I20260620 18:42:39.027639 14131 leader_election.cc:288] T afd148b67e7144d9bfb380e47a1bf33b P 789b964919a043e6a2b7130b340139af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06416c21938742de87f05ca67fcc42dc (127.5.225.129:44023), 3cf75338fddf4a90b0628d178afd67f1 (127.5.225.130:39239)
I20260620 18:42:39.028177 13763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afd148b67e7144d9bfb380e47a1bf33b" candidate_uuid: "789b964919a043e6a2b7130b340139af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06416c21938742de87f05ca67fcc42dc" is_pre_election: true
I20260620 18:42:39.028465 13763 raft_consensus.cc:2454] T afd148b67e7144d9bfb380e47a1bf33b P 06416c21938742de87f05ca67fcc42dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 789b964919a043e6a2b7130b340139af in term 0.
I20260620 18:42:39.028426 13902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afd148b67e7144d9bfb380e47a1bf33b" candidate_uuid: "789b964919a043e6a2b7130b340139af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cf75338fddf4a90b0628d178afd67f1" is_pre_election: true
I20260620 18:42:39.028657 13902 raft_consensus.cc:2454] T afd148b67e7144d9bfb380e47a1bf33b P 3cf75338fddf4a90b0628d178afd67f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 789b964919a043e6a2b7130b340139af in term 0.
I20260620 18:42:39.028981 13970 leader_election.cc:302] T afd148b67e7144d9bfb380e47a1bf33b P 789b964919a043e6a2b7130b340139af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06416c21938742de87f05ca67fcc42dc, 789b964919a043e6a2b7130b340139af; no voters:
I20260620 18:42:39.029238 14131 raft_consensus.cc:2781] T afd148b67e7144d9bfb380e47a1bf33b P 789b964919a043e6a2b7130b340139af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:42:39.029384 14131 raft_consensus.cc:482] T afd148b67e7144d9bfb380e47a1bf33b P 789b964919a043e6a2b7130b340139af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:42:39.029500 14131 raft_consensus.cc:3037] T afd148b67e7144d9bfb380e47a1bf33b P 789b964919a043e6a2b7130b340139af [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:39.032310 14131 raft_consensus.cc:504] T afd148b67e7144d9bfb380e47a1bf33b P 789b964919a043e6a2b7130b340139af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } }
I20260620 18:42:39.032914 14131 leader_election.cc:288] T afd148b67e7144d9bfb380e47a1bf33b P 789b964919a043e6a2b7130b340139af [CANDIDATE]: Term 1 election: Requested vote from peers 06416c21938742de87f05ca67fcc42dc (127.5.225.129:44023), 3cf75338fddf4a90b0628d178afd67f1 (127.5.225.130:39239)
I20260620 18:42:39.033414 13902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afd148b67e7144d9bfb380e47a1bf33b" candidate_uuid: "789b964919a043e6a2b7130b340139af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cf75338fddf4a90b0628d178afd67f1"
I20260620 18:42:39.033407 13763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afd148b67e7144d9bfb380e47a1bf33b" candidate_uuid: "789b964919a043e6a2b7130b340139af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06416c21938742de87f05ca67fcc42dc"
I20260620 18:42:39.033612 13902 raft_consensus.cc:3037] T afd148b67e7144d9bfb380e47a1bf33b P 3cf75338fddf4a90b0628d178afd67f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:39.033627 13763 raft_consensus.cc:3037] T afd148b67e7144d9bfb380e47a1bf33b P 06416c21938742de87f05ca67fcc42dc [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:39.035661 13902 raft_consensus.cc:2454] T afd148b67e7144d9bfb380e47a1bf33b P 3cf75338fddf4a90b0628d178afd67f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 789b964919a043e6a2b7130b340139af in term 1.
I20260620 18:42:39.036057 13967 leader_election.cc:302] T afd148b67e7144d9bfb380e47a1bf33b P 789b964919a043e6a2b7130b340139af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cf75338fddf4a90b0628d178afd67f1, 789b964919a043e6a2b7130b340139af; no voters:
I20260620 18:42:39.036346 14131 raft_consensus.cc:2781] T afd148b67e7144d9bfb380e47a1bf33b P 789b964919a043e6a2b7130b340139af [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:42:39.036504 14131 raft_consensus.cc:686] T afd148b67e7144d9bfb380e47a1bf33b P 789b964919a043e6a2b7130b340139af [term 1 LEADER]: Becoming Leader. State: Replica: 789b964919a043e6a2b7130b340139af, State: Running, Role: LEADER
I20260620 18:42:39.036667 13763 raft_consensus.cc:2454] T afd148b67e7144d9bfb380e47a1bf33b P 06416c21938742de87f05ca67fcc42dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 789b964919a043e6a2b7130b340139af in term 1.
I20260620 18:42:39.036800 14131 consensus_queue.cc:237] T afd148b67e7144d9bfb380e47a1bf33b P 789b964919a043e6a2b7130b340139af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } }
I20260620 18:42:39.039559 13600 catalog_manager.cc:5495] T afd148b67e7144d9bfb380e47a1bf33b P 789b964919a043e6a2b7130b340139af reported cstate change: term changed from 0 to 1, leader changed from <none> to 789b964919a043e6a2b7130b340139af (127.5.225.131). New cstate: current_term: 1 leader_uuid: "789b964919a043e6a2b7130b340139af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } health_report { overall_health: HEALTHY } } }
Jun 20 18:42:39 dist-test-slave-k6lg krb5kdc[13556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980955, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.225.190@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM
I20260620 18:42:39.095481 14147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:39.080753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:40433 (local address 127.0.0.1:38550)
0620 18:42:39.081444 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:39.081463 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:39.081594 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:39.081926 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:39.081930 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:39.082717 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:39.082963 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:39.082972 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:39.084380 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:39.084383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:39.085347 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:39.085357 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:39.086319 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:39.086323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:39.086449 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:39.087121 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:39.087137 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:39.089418 (+ 2281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:39.093202 (+ 3784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:39.093211 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:39.093216 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:39.093605 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:39.093974 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:39.093977 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:39.093979 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:39.094114 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:39.094713 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:39.094719 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:39.094865 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 18:42:39.094997 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 18:42:39.095185 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":405,"threads_started":1}
I20260620 18:42:39.095803 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:39.080667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38550 (local address 127.5.225.190:40433)
0620 18:42:39.080875 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:39.080882 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:39.081557 (+ 675us) server_negotiation.cc:407] Connection header received
0620 18:42:39.081694 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:39.081700 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:39.081781 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:39.081896 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:39.083102 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:39.084250 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:39.085489 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:39.086126 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:39.086291 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:39.089612 (+ 3321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:39.089618 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:39.089625 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:39.089663 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:39.093083 (+ 3420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:39.093722 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:39.093727 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:39.093730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:39.093866 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:39.094239 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:39.094244 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:39.094248 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:39.094581 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:39.094749 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:39.095129 (+ 380us) server_negotiation.cc:299] Negotiation successful
0620 18:42:39.095349 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 18:42:39.100659 13600 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:38550:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 18:42:39.101684 13600 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:42:39.108278 13738 tablet_service.cc:1467] Processing CreateTablet for tablet 404607a853d3465e886d209c2ea01ee5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2b0313820d76467198e4e3168f5f7a1e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:42:39.108386 13877 tablet_service.cc:1467] Processing CreateTablet for tablet 404607a853d3465e886d209c2ea01ee5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2b0313820d76467198e4e3168f5f7a1e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:42:39.108824 13738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 404607a853d3465e886d209c2ea01ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:39.108946 13877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 404607a853d3465e886d209c2ea01ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:39.109073 14020 tablet_service.cc:1467] Processing CreateTablet for tablet 404607a853d3465e886d209c2ea01ee5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2b0313820d76467198e4e3168f5f7a1e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:42:39.109588 14020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 404607a853d3465e886d209c2ea01ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:39.116096 14115 tablet_bootstrap.cc:492] T 404607a853d3465e886d209c2ea01ee5 P 3cf75338fddf4a90b0628d178afd67f1: Bootstrap starting.
I20260620 18:42:39.116587 14112 tablet_bootstrap.cc:492] T 404607a853d3465e886d209c2ea01ee5 P 789b964919a043e6a2b7130b340139af: Bootstrap starting.
I20260620 18:42:39.117790 14115 tablet_bootstrap.cc:654] T 404607a853d3465e886d209c2ea01ee5 P 3cf75338fddf4a90b0628d178afd67f1: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:39.118010 14112 tablet_bootstrap.cc:654] T 404607a853d3465e886d209c2ea01ee5 P 789b964919a043e6a2b7130b340139af: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:39.118995 14111 tablet_bootstrap.cc:492] T 404607a853d3465e886d209c2ea01ee5 P 06416c21938742de87f05ca67fcc42dc: Bootstrap starting.
I20260620 18:42:39.120623 14111 tablet_bootstrap.cc:654] T 404607a853d3465e886d209c2ea01ee5 P 06416c21938742de87f05ca67fcc42dc: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:39.122087 14115 tablet_bootstrap.cc:492] T 404607a853d3465e886d209c2ea01ee5 P 3cf75338fddf4a90b0628d178afd67f1: No bootstrap required, opened a new log
I20260620 18:42:39.122268 14115 ts_tablet_manager.cc:1397] T 404607a853d3465e886d209c2ea01ee5 P 3cf75338fddf4a90b0628d178afd67f1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 18:42:39.122977 14115 raft_consensus.cc:348] T 404607a853d3465e886d209c2ea01ee5 P 3cf75338fddf4a90b0628d178afd67f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } }
I20260620 18:42:39.123217 14115 raft_consensus.cc:374] T 404607a853d3465e886d209c2ea01ee5 P 3cf75338fddf4a90b0628d178afd67f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:39.123294 14115 raft_consensus.cc:729] T 404607a853d3465e886d209c2ea01ee5 P 3cf75338fddf4a90b0628d178afd67f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cf75338fddf4a90b0628d178afd67f1, State: Initialized, Role: FOLLOWER
I20260620 18:42:39.123562 14115 consensus_queue.cc:260] T 404607a853d3465e886d209c2ea01ee5 P 3cf75338fddf4a90b0628d178afd67f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } }
I20260620 18:42:39.124009 14119 tablet_replica.cc:439] T 404607a853d3465e886d209c2ea01ee5 P 3cf75338fddf4a90b0628d178afd67f1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } } }
I20260620 18:42:39.124354 14119 tablet_replica.cc:442] T 404607a853d3465e886d209c2ea01ee5 P 3cf75338fddf4a90b0628d178afd67f1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:42:39.124993 14115 ts_tablet_manager.cc:1428] T 404607a853d3465e886d209c2ea01ee5 P 3cf75338fddf4a90b0628d178afd67f1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 18:42:39.129755 14112 tablet_bootstrap.cc:492] T 404607a853d3465e886d209c2ea01ee5 P 789b964919a043e6a2b7130b340139af: No bootstrap required, opened a new log
I20260620 18:42:39.129940 14112 ts_tablet_manager.cc:1397] T 404607a853d3465e886d209c2ea01ee5 P 789b964919a043e6a2b7130b340139af: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.007s
I20260620 18:42:39.130882 14112 raft_consensus.cc:348] T 404607a853d3465e886d209c2ea01ee5 P 789b964919a043e6a2b7130b340139af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } }
I20260620 18:42:39.131177 14112 raft_consensus.cc:374] T 404607a853d3465e886d209c2ea01ee5 P 789b964919a043e6a2b7130b340139af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:39.131294 14112 raft_consensus.cc:729] T 404607a853d3465e886d209c2ea01ee5 P 789b964919a043e6a2b7130b340139af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789b964919a043e6a2b7130b340139af, State: Initialized, Role: FOLLOWER
I20260620 18:42:39.131628 14111 tablet_bootstrap.cc:492] T 404607a853d3465e886d209c2ea01ee5 P 06416c21938742de87f05ca67fcc42dc: No bootstrap required, opened a new log
I20260620 18:42:39.131563 14112 consensus_queue.cc:260] T 404607a853d3465e886d209c2ea01ee5 P 789b964919a043e6a2b7130b340139af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } }
I20260620 18:42:39.131824 14111 ts_tablet_manager.cc:1397] T 404607a853d3465e886d209c2ea01ee5 P 06416c21938742de87f05ca67fcc42dc: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s
I20260620 18:42:39.131987 14131 tablet_replica.cc:439] T 404607a853d3465e886d209c2ea01ee5 P 789b964919a043e6a2b7130b340139af: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } } }
I20260620 18:42:39.132181 14112 ts_tablet_manager.cc:1428] T 404607a853d3465e886d209c2ea01ee5 P 789b964919a043e6a2b7130b340139af: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:42:39.132284 14131 tablet_replica.cc:442] T 404607a853d3465e886d209c2ea01ee5 P 789b964919a043e6a2b7130b340139af: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:42:39.133072 14111 raft_consensus.cc:348] T 404607a853d3465e886d209c2ea01ee5 P 06416c21938742de87f05ca67fcc42dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } }
I20260620 18:42:39.133293 14111 raft_consensus.cc:374] T 404607a853d3465e886d209c2ea01ee5 P 06416c21938742de87f05ca67fcc42dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:39.133358 14111 raft_consensus.cc:729] T 404607a853d3465e886d209c2ea01ee5 P 06416c21938742de87f05ca67fcc42dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06416c21938742de87f05ca67fcc42dc, State: Initialized, Role: FOLLOWER
I20260620 18:42:39.133611 14111 consensus_queue.cc:260] T 404607a853d3465e886d209c2ea01ee5 P 06416c21938742de87f05ca67fcc42dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } }
I20260620 18:42:39.134011 14117 tablet_replica.cc:439] T 404607a853d3465e886d209c2ea01ee5 P 06416c21938742de87f05ca67fcc42dc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } } }
I20260620 18:42:39.134356 14117 tablet_replica.cc:442] T 404607a853d3465e886d209c2ea01ee5 P 06416c21938742de87f05ca67fcc42dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:42:39.134893 14111 ts_tablet_manager.cc:1428] T 404607a853d3465e886d209c2ea01ee5 P 06416c21938742de87f05ca67fcc42dc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:42:39.269666 14117 raft_consensus.cc:482] T 404607a853d3465e886d209c2ea01ee5 P 06416c21938742de87f05ca67fcc42dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:42:39.269933 14117 raft_consensus.cc:504] T 404607a853d3465e886d209c2ea01ee5 P 06416c21938742de87f05ca67fcc42dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } }
I20260620 18:42:39.271764 14117 leader_election.cc:288] T 404607a853d3465e886d209c2ea01ee5 P 06416c21938742de87f05ca67fcc42dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cf75338fddf4a90b0628d178afd67f1 (127.5.225.130:39239), 789b964919a043e6a2b7130b340139af (127.5.225.131:37959)
I20260620 18:42:39.281376 14106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:39.272678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:34661 (local address 127.5.225.131:37959)
0620 18:42:39.272857 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:39.272864 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:39.273754 (+ 890us) server_negotiation.cc:407] Connection header received
0620 18:42:39.273954 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:39.273958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:39.274032 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:39.274127 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:39.274913 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:39.276041 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:39.279036 (+ 2995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:39.280316 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:39.280479 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:39.280573 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:39.280906 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 18:42:39.281052 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 18:42:39.281473 14151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:39.272042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.131:37959 (local address 127.5.225.129:34661)
0620 18:42:39.273639 (+ 1597us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:39.273656 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:39.273806 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:39.274191 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:39.274196 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:39.274242 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:39.274714 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:39.274734 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:39.276212 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:39.276220 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:39.278860 (+ 2640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:39.278875 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:39.280570 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:39.280575 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:39.280738 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:39.280772 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 18:42:39.280882 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 18:42:39.281018 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1460,"mutex_wait_us":157,"thread_start_us":1222,"threads_started":1}
I20260620 18:42:39.282402 14041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "404607a853d3465e886d209c2ea01ee5" candidate_uuid: "06416c21938742de87f05ca67fcc42dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789b964919a043e6a2b7130b340139af" is_pre_election: true
I20260620 18:42:39.282941 14041 raft_consensus.cc:2454] T 404607a853d3465e886d209c2ea01ee5 P 789b964919a043e6a2b7130b340139af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06416c21938742de87f05ca67fcc42dc in term 0.
I20260620 18:42:39.283293 14152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:39.272042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.130:39239 (local address 127.5.225.129:40339)
0620 18:42:39.273336 (+ 1294us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:39.273364 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:39.273588 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:39.274310 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:39.274314 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:39.274358 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:39.274776 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:39.274786 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:39.276266 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:39.276272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:39.278860 (+ 2588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:39.278875 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:39.282443 (+ 3568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:39.282451 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:39.282623 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:39.282658 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 18:42:39.282774 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 18:42:39.282902 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1106,"mutex_wait_us":156,"spinlock_wait_cycles":896,"thread_start_us":907,"threads_started":1}
I20260620 18:42:39.283646 14109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:39.272450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:40339 (local address 127.5.225.130:39239)
0620 18:42:39.272646 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:39.272653 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:39.273463 (+ 810us) server_negotiation.cc:407] Connection header received
0620 18:42:39.273741 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:39.273746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:39.273824 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:39.273924 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:39.275027 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:39.276144 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:39.280670 (+ 4526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:39.281994 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:39.282189 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:39.282264 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:39.282885 (+ 621us) server_negotiation.cc:299] Negotiation successful
0620 18:42:39.283030 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 18:42:39.283541 13689 leader_election.cc:302] T 404607a853d3465e886d209c2ea01ee5 P 06416c21938742de87f05ca67fcc42dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06416c21938742de87f05ca67fcc42dc, 789b964919a043e6a2b7130b340139af; no voters:
I20260620 18:42:39.283944 14117 raft_consensus.cc:2781] T 404607a853d3465e886d209c2ea01ee5 P 06416c21938742de87f05ca67fcc42dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:42:39.284070 14117 raft_consensus.cc:482] T 404607a853d3465e886d209c2ea01ee5 P 06416c21938742de87f05ca67fcc42dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:42:39.284186 14117 raft_consensus.cc:3037] T 404607a853d3465e886d209c2ea01ee5 P 06416c21938742de87f05ca67fcc42dc [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:39.286338 14117 raft_consensus.cc:504] T 404607a853d3465e886d209c2ea01ee5 P 06416c21938742de87f05ca67fcc42dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } }
I20260620 18:42:39.286759 13902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "404607a853d3465e886d209c2ea01ee5" candidate_uuid: "06416c21938742de87f05ca67fcc42dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cf75338fddf4a90b0628d178afd67f1" is_pre_election: true
I20260620 18:42:39.287020 14117 leader_election.cc:288] T 404607a853d3465e886d209c2ea01ee5 P 06416c21938742de87f05ca67fcc42dc [CANDIDATE]: Term 1 election: Requested vote from peers 3cf75338fddf4a90b0628d178afd67f1 (127.5.225.130:39239), 789b964919a043e6a2b7130b340139af (127.5.225.131:37959)
I20260620 18:42:39.287014 13902 raft_consensus.cc:2454] T 404607a853d3465e886d209c2ea01ee5 P 3cf75338fddf4a90b0628d178afd67f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06416c21938742de87f05ca67fcc42dc in term 0.
I20260620 18:42:39.287518 13902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "404607a853d3465e886d209c2ea01ee5" candidate_uuid: "06416c21938742de87f05ca67fcc42dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cf75338fddf4a90b0628d178afd67f1"
I20260620 18:42:39.287611 14041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "404607a853d3465e886d209c2ea01ee5" candidate_uuid: "06416c21938742de87f05ca67fcc42dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789b964919a043e6a2b7130b340139af"
I20260620 18:42:39.287724 13902 raft_consensus.cc:3037] T 404607a853d3465e886d209c2ea01ee5 P 3cf75338fddf4a90b0628d178afd67f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:39.287815 14041 raft_consensus.cc:3037] T 404607a853d3465e886d209c2ea01ee5 P 789b964919a043e6a2b7130b340139af [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:39.289927 14041 raft_consensus.cc:2454] T 404607a853d3465e886d209c2ea01ee5 P 789b964919a043e6a2b7130b340139af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06416c21938742de87f05ca67fcc42dc in term 1.
I20260620 18:42:39.290268 13689 leader_election.cc:302] T 404607a853d3465e886d209c2ea01ee5 P 06416c21938742de87f05ca67fcc42dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06416c21938742de87f05ca67fcc42dc, 789b964919a043e6a2b7130b340139af; no voters:
I20260620 18:42:39.290514 14117 raft_consensus.cc:2781] T 404607a853d3465e886d209c2ea01ee5 P 06416c21938742de87f05ca67fcc42dc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:42:39.290614 13902 raft_consensus.cc:2454] T 404607a853d3465e886d209c2ea01ee5 P 3cf75338fddf4a90b0628d178afd67f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06416c21938742de87f05ca67fcc42dc in term 1.
I20260620 18:42:39.291091 14117 raft_consensus.cc:686] T 404607a853d3465e886d209c2ea01ee5 P 06416c21938742de87f05ca67fcc42dc [term 1 LEADER]: Becoming Leader. State: Replica: 06416c21938742de87f05ca67fcc42dc, State: Running, Role: LEADER
I20260620 18:42:39.291730 14117 consensus_queue.cc:237] T 404607a853d3465e886d209c2ea01ee5 P 06416c21938742de87f05ca67fcc42dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } }
I20260620 18:42:39.293728 14153 tablet_replica.cc:439] T 404607a853d3465e886d209c2ea01ee5 P 06416c21938742de87f05ca67fcc42dc: TxnStatusTablet state changed. Reason: New leader 06416c21938742de87f05ca67fcc42dc. Latest consensus state: current_term: 1 leader_uuid: "06416c21938742de87f05ca67fcc42dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } } }
I20260620 18:42:39.294122 14153 tablet_replica.cc:442] T 404607a853d3465e886d209c2ea01ee5 P 06416c21938742de87f05ca67fcc42dc: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:42:39.294796 14155 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 18:42:39.295009 14155 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 18:42:39.295516 13599 catalog_manager.cc:5495] T 404607a853d3465e886d209c2ea01ee5 P 06416c21938742de87f05ca67fcc42dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 06416c21938742de87f05ca67fcc42dc (127.5.225.129). New cstate: current_term: 1 leader_uuid: "06416c21938742de87f05ca67fcc42dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20260620 18:42:39.329460 14147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:39.320973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:44023 (local address 127.0.0.1:57872)
0620 18:42:39.321215 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:39.321236 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:39.321369 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:39.323100 (+ 1731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:39.323106 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:39.323120 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:42:39.323349 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:39.323362 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:39.324909 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:39.324914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:39.326539 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:39.326556 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:39.327534 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:39.327541 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:39.327746 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:39.327820 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:42:39.328912 (+ 1092us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:42:39.328934 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 18:42:39.328994 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 18:42:39.329147 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 18:42:39.330309 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:39.321378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57872 (local address 127.5.225.129:44023)
0620 18:42:39.322601 (+ 1223us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:39.322607 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:39.322626 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:42:39.322733 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:39.322738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:39.322868 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:39.323065 (+ 197us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:42:39.323512 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:39.324767 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:39.326709 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:39.327374 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:39.327493 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:39.328030 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:42:39.328779 (+ 749us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:42:39.329766 (+ 987us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:39.329822 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 18:42:39.329913 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1103,"thread_start_us":946,"threads_started":1}
I20260620 18:42:39.334913 14041 raft_consensus.cc:1261] T 404607a853d3465e886d209c2ea01ee5 P 789b964919a043e6a2b7130b340139af [term 1 FOLLOWER]: Refusing update from remote peer 06416c21938742de87f05ca67fcc42dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:42:39.335644 13902 raft_consensus.cc:1261] T 404607a853d3465e886d209c2ea01ee5 P 3cf75338fddf4a90b0628d178afd67f1 [term 1 FOLLOWER]: Refusing update from remote peer 06416c21938742de87f05ca67fcc42dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:42:39.335796 14117 consensus_queue.cc:1035] T 404607a853d3465e886d209c2ea01ee5 P 06416c21938742de87f05ca67fcc42dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:42:39.336228 14153 consensus_queue.cc:1035] T 404607a853d3465e886d209c2ea01ee5 P 06416c21938742de87f05ca67fcc42dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:42:39.349677 14119 tablet_replica.cc:439] T 404607a853d3465e886d209c2ea01ee5 P 3cf75338fddf4a90b0628d178afd67f1: TxnStatusTablet state changed. Reason: New leader 06416c21938742de87f05ca67fcc42dc. Latest consensus state: current_term: 1 leader_uuid: "06416c21938742de87f05ca67fcc42dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } } }
I20260620 18:42:39.349994 14119 tablet_replica.cc:442] T 404607a853d3465e886d209c2ea01ee5 P 3cf75338fddf4a90b0628d178afd67f1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:42:39.354436 14117 tablet_replica.cc:439] T 404607a853d3465e886d209c2ea01ee5 P 06416c21938742de87f05ca67fcc42dc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "06416c21938742de87f05ca67fcc42dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } } }
I20260620 18:42:39.354709 14117 tablet_replica.cc:442] T 404607a853d3465e886d209c2ea01ee5 P 06416c21938742de87f05ca67fcc42dc: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:42:39.354885 14127 mvcc.cc:205] Tried to move back new op lower bound from 7298994009427341312 to 7298994009262669824. Current Snapshot: MvccSnapshot[applied={T|T < 7298994009427341312}]
I20260620 18:42:39.355261 14119 tablet_replica.cc:439] T 404607a853d3465e886d209c2ea01ee5 P 3cf75338fddf4a90b0628d178afd67f1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "06416c21938742de87f05ca67fcc42dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } } }
I20260620 18:42:39.355602 14119 tablet_replica.cc:442] T 404607a853d3465e886d209c2ea01ee5 P 3cf75338fddf4a90b0628d178afd67f1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:42:39.361752 14131 tablet_replica.cc:439] T 404607a853d3465e886d209c2ea01ee5 P 789b964919a043e6a2b7130b340139af: TxnStatusTablet state changed. Reason: New leader 06416c21938742de87f05ca67fcc42dc. Latest consensus state: current_term: 1 leader_uuid: "06416c21938742de87f05ca67fcc42dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } } }
I20260620 18:42:39.362043 14131 tablet_replica.cc:442] T 404607a853d3465e886d209c2ea01ee5 P 789b964919a043e6a2b7130b340139af: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:42:39.363267 14117 tablet_replica.cc:439] T 404607a853d3465e886d209c2ea01ee5 P 06416c21938742de87f05ca67fcc42dc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "06416c21938742de87f05ca67fcc42dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } } }
I20260620 18:42:39.363511 14117 tablet_replica.cc:442] T 404607a853d3465e886d209c2ea01ee5 P 06416c21938742de87f05ca67fcc42dc: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:42:39.365947 14131 tablet_replica.cc:439] T 404607a853d3465e886d209c2ea01ee5 P 789b964919a043e6a2b7130b340139af: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "06416c21938742de87f05ca67fcc42dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } } }
I20260620 18:42:39.366264 14131 tablet_replica.cc:442] T 404607a853d3465e886d209c2ea01ee5 P 789b964919a043e6a2b7130b340139af: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:42:39.377627 14163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:39.371868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:44023 (local address 127.5.225.131:37279)
0620 18:42:39.372489 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:39.372505 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:39.372644 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:39.372967 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:39.372970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:39.372979 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:42:39.373221 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:39.373228 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:39.374660 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:39.374664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:39.375634 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:39.375644 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:39.376542 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:39.376546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:39.376746 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:39.376840 (+ 94us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:42:39.377210 (+ 370us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:42:39.377217 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 18:42:39.377273 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 18:42:39.377371 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":329,"threads_started":1}
I20260620 18:42:39.377664 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:39.371873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:37279 (local address 127.5.225.129:44023)
0620 18:42:39.372039 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:39.372046 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:39.372643 (+ 597us) server_negotiation.cc:407] Connection header received
0620 18:42:39.372771 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:39.372777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:39.372856 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:39.372956 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:42:39.373348 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:39.374544 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:39.375772 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:39.376412 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:39.376530 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:39.376994 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:42:39.377116 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:42:39.377203 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:39.377316 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 18:42:39.377392 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 18:42:39.388780 13901 raft_consensus.cc:1261] T afd148b67e7144d9bfb380e47a1bf33b P 3cf75338fddf4a90b0628d178afd67f1 [term 1 FOLLOWER]: Refusing update from remote peer 789b964919a043e6a2b7130b340139af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:42:39.389062 13763 raft_consensus.cc:1261] T afd148b67e7144d9bfb380e47a1bf33b P 06416c21938742de87f05ca67fcc42dc [term 1 FOLLOWER]: Refusing update from remote peer 789b964919a043e6a2b7130b340139af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:42:39.389354 14118 consensus_queue.cc:1035] T afd148b67e7144d9bfb380e47a1bf33b P 789b964919a043e6a2b7130b340139af [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:42:39.390141 14118 consensus_queue.cc:1035] T afd148b67e7144d9bfb380e47a1bf33b P 789b964919a043e6a2b7130b340139af [LEADER]: Connected to new peer: Peer: permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:42:39.425803 14106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:39.419852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:38197 (local address 127.5.225.131:37959)
0620 18:42:39.420002 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:39.420006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:39.420251 (+ 245us) server_negotiation.cc:407] Connection header received
0620 18:42:39.420440 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:39.420443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:39.420499 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:39.420570 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:42:39.421164 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:39.421979 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:39.424136 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:39.424514 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:39.424606 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:39.425091 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:42:39.425187 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:42:39.425254 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:39.425425 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 18:42:39.425501 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 18:42:39.425954 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:39.419563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.131:37959 (local address 127.5.225.129:38197)
0620 18:42:39.420140 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:39.420160 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:39.420303 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:39.420613 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:39.420620 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:39.420633 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:42:39.421022 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:39.421034 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:39.422098 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:39.422104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:39.424009 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:39.424024 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:39.424682 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:39.424689 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:39.424862 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:39.424937 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:42:39.425286 (+ 349us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:42:39.425303 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 18:42:39.425387 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 18:42:39.425526 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":531,"threads_started":1}
I20260620 18:42:39.463290 13599 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:42:39.463531 13599 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:42:39.466760 13599 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 972f20065ea9422a838127c325ed3908: Sending DeleteTablet for 3 replicas of tablet afd148b67e7144d9bfb380e47a1bf33b
I20260620 18:42:39.467433 13738 tablet_service.cc:1514] Processing DeleteTablet for tablet afd148b67e7144d9bfb380e47a1bf33b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:42:39 UTC) from {username='oryx', principal='oryx/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:57858
I20260620 18:42:39.467679 14020 tablet_service.cc:1514] Processing DeleteTablet for tablet afd148b67e7144d9bfb380e47a1bf33b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:42:39 UTC) from {username='oryx', principal='oryx/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:57452
I20260620 18:42:39.467679 13877 tablet_service.cc:1514] Processing DeleteTablet for tablet afd148b67e7144d9bfb380e47a1bf33b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:42:39 UTC) from {username='oryx', principal='oryx/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:46782
I20260620 18:42:39.478317 14173 tablet_replica.cc:330] T afd148b67e7144d9bfb380e47a1bf33b P 06416c21938742de87f05ca67fcc42dc: stopping tablet replica
I20260620 18:42:39.478622 14173 raft_consensus.cc:2229] T afd148b67e7144d9bfb380e47a1bf33b P 06416c21938742de87f05ca67fcc42dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:42:39.478991 14173 raft_consensus.cc:2258] T afd148b67e7144d9bfb380e47a1bf33b P 06416c21938742de87f05ca67fcc42dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:42:39.480655 14173 ts_tablet_manager.cc:1905] T afd148b67e7144d9bfb380e47a1bf33b P 06416c21938742de87f05ca67fcc42dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:42:39.492300 14173 ts_tablet_manager.cc:1918] T afd148b67e7144d9bfb380e47a1bf33b P 06416c21938742de87f05ca67fcc42dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:42:39.492496 14173 log.cc:1196] T afd148b67e7144d9bfb380e47a1bf33b P 06416c21938742de87f05ca67fcc42dc: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-0/wal/wals/afd148b67e7144d9bfb380e47a1bf33b
I20260620 18:42:39.493101 14173 ts_tablet_manager.cc:1939] T afd148b67e7144d9bfb380e47a1bf33b P 06416c21938742de87f05ca67fcc42dc: Deleting consensus metadata
I20260620 18:42:39.494578 13588 catalog_manager.cc:4841] TS 06416c21938742de87f05ca67fcc42dc (127.5.225.129:44023): tablet afd148b67e7144d9bfb380e47a1bf33b (table test-table [id=e7bf7528287e47afb06a80bfc98572e3]) successfully deleted
I20260620 18:42:39.499845 14174 tablet_replica.cc:330] T afd148b67e7144d9bfb380e47a1bf33b P 3cf75338fddf4a90b0628d178afd67f1: stopping tablet replica
I20260620 18:42:39.500777 14174 raft_consensus.cc:2229] T afd148b67e7144d9bfb380e47a1bf33b P 3cf75338fddf4a90b0628d178afd67f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:42:39.512770 14174 raft_consensus.cc:2258] T afd148b67e7144d9bfb380e47a1bf33b P 3cf75338fddf4a90b0628d178afd67f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:42:39.516958 14175 tablet_replica.cc:330] T afd148b67e7144d9bfb380e47a1bf33b P 789b964919a043e6a2b7130b340139af: stopping tablet replica
I20260620 18:42:39.517280 14175 raft_consensus.cc:2229] T afd148b67e7144d9bfb380e47a1bf33b P 789b964919a043e6a2b7130b340139af [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:42:39.517722 14174 ts_tablet_manager.cc:1905] T afd148b67e7144d9bfb380e47a1bf33b P 3cf75338fddf4a90b0628d178afd67f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:42:39.517863 14175 raft_consensus.cc:2258] T afd148b67e7144d9bfb380e47a1bf33b P 789b964919a043e6a2b7130b340139af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:42:39.518980 14175 ts_tablet_manager.cc:1905] T afd148b67e7144d9bfb380e47a1bf33b P 789b964919a043e6a2b7130b340139af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:42:39.524726 14174 ts_tablet_manager.cc:1918] T afd148b67e7144d9bfb380e47a1bf33b P 3cf75338fddf4a90b0628d178afd67f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:42:39.524856 14174 log.cc:1196] T afd148b67e7144d9bfb380e47a1bf33b P 3cf75338fddf4a90b0628d178afd67f1: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-1/wal/wals/afd148b67e7144d9bfb380e47a1bf33b
I20260620 18:42:39.525274 14174 ts_tablet_manager.cc:1939] T afd148b67e7144d9bfb380e47a1bf33b P 3cf75338fddf4a90b0628d178afd67f1: Deleting consensus metadata
I20260620 18:42:39.525647 14175 ts_tablet_manager.cc:1918] T afd148b67e7144d9bfb380e47a1bf33b P 789b964919a043e6a2b7130b340139af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:42:39.525774 14175 log.cc:1196] T afd148b67e7144d9bfb380e47a1bf33b P 789b964919a043e6a2b7130b340139af: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-2/wal/wals/afd148b67e7144d9bfb380e47a1bf33b
I20260620 18:42:39.526549 13585 catalog_manager.cc:4841] TS 3cf75338fddf4a90b0628d178afd67f1 (127.5.225.130:39239): tablet afd148b67e7144d9bfb380e47a1bf33b (table test-table [id=e7bf7528287e47afb06a80bfc98572e3]) successfully deleted
I20260620 18:42:39.527061 14175 ts_tablet_manager.cc:1939] T afd148b67e7144d9bfb380e47a1bf33b P 789b964919a043e6a2b7130b340139af: Deleting consensus metadata
I20260620 18:42:39.528195 13586 catalog_manager.cc:4841] TS 789b964919a043e6a2b7130b340139af (127.5.225.131:37959): tablet afd148b67e7144d9bfb380e47a1bf33b (table test-table [id=e7bf7528287e47afb06a80bfc98572e3]) successfully deleted
Jun 20 18:42:39 dist-test-slave-k6lg krb5kdc[13556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 18:42:39 dist-test-slave-k6lg krb5kdc[13556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:42:39 dist-test-slave-k6lg krb5kdc[13556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 18:42:39.572913 13696 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:39.560306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38554 (local address 127.5.225.190:40433)
0620 18:42:39.560561 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:39.560568 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:39.560942 (+ 374us) server_negotiation.cc:407] Connection header received
0620 18:42:39.561151 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:39.561155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:39.561230 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:39.561318 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:39.563072 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:39.563786 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:39.565613 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:39.566105 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:39.566269 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:39.572638 (+ 6369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38554: BlockingRecv error: recv got EOF from 127.0.0.1:38554 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Jun 20 18:42:39 dist-test-slave-k6lg krb5kdc[13556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM
I20260620 18:42:39.591531 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:39.575908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38564 (local address 127.5.225.190:40433)
0620 18:42:39.576106 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:39.576112 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:39.576554 (+ 442us) server_negotiation.cc:407] Connection header received
0620 18:42:39.576751 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:39.576756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:39.576840 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:39.576949 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:39.578104 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:39.579172 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:39.580599 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:39.581283 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:39.581395 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:39.585445 (+ 4050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:39.585452 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:39.585457 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:39.585497 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:39.588731 (+ 3234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:39.589401 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:39.589406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:39.589410 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:39.589584 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:39.589943 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:39.589948 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:39.589951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:39.590316 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:39.590582 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:39.590939 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 18:42:39.591164 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:42:39.595259 13599 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:42:39.595954 13599 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:42:39.603065 13738 tablet_service.cc:1467] Processing CreateTablet for tablet f068a4ffdeb547ea97c691b575b5e0af (DEFAULT_TABLE table=test-table [id=c36b4f5c11cc43aab31246e6619620f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:42:39.603120 13877 tablet_service.cc:1467] Processing CreateTablet for tablet f068a4ffdeb547ea97c691b575b5e0af (DEFAULT_TABLE table=test-table [id=c36b4f5c11cc43aab31246e6619620f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:42:39.603561 13877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f068a4ffdeb547ea97c691b575b5e0af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:39.603570 13738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f068a4ffdeb547ea97c691b575b5e0af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:39.603621 14020 tablet_service.cc:1467] Processing CreateTablet for tablet f068a4ffdeb547ea97c691b575b5e0af (DEFAULT_TABLE table=test-table [id=c36b4f5c11cc43aab31246e6619620f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:42:39.603992 14020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f068a4ffdeb547ea97c691b575b5e0af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:39.610062 14115 tablet_bootstrap.cc:492] T f068a4ffdeb547ea97c691b575b5e0af P 3cf75338fddf4a90b0628d178afd67f1: Bootstrap starting.
I20260620 18:42:39.611752 14115 tablet_bootstrap.cc:654] T f068a4ffdeb547ea97c691b575b5e0af P 3cf75338fddf4a90b0628d178afd67f1: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:39.612802 14111 tablet_bootstrap.cc:492] T f068a4ffdeb547ea97c691b575b5e0af P 06416c21938742de87f05ca67fcc42dc: Bootstrap starting.
I20260620 18:42:39.614106 14111 tablet_bootstrap.cc:654] T f068a4ffdeb547ea97c691b575b5e0af P 06416c21938742de87f05ca67fcc42dc: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:39.615739 14112 tablet_bootstrap.cc:492] T f068a4ffdeb547ea97c691b575b5e0af P 789b964919a043e6a2b7130b340139af: Bootstrap starting.
I20260620 18:42:39.617309 14112 tablet_bootstrap.cc:654] T f068a4ffdeb547ea97c691b575b5e0af P 789b964919a043e6a2b7130b340139af: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:39.618952 14115 tablet_bootstrap.cc:492] T f068a4ffdeb547ea97c691b575b5e0af P 3cf75338fddf4a90b0628d178afd67f1: No bootstrap required, opened a new log
I20260620 18:42:39.619105 14115 ts_tablet_manager.cc:1397] T f068a4ffdeb547ea97c691b575b5e0af P 3cf75338fddf4a90b0628d178afd67f1: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20260620 18:42:39.619925 14115 raft_consensus.cc:348] T f068a4ffdeb547ea97c691b575b5e0af P 3cf75338fddf4a90b0628d178afd67f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } }
I20260620 18:42:39.620177 14115 raft_consensus.cc:374] T f068a4ffdeb547ea97c691b575b5e0af P 3cf75338fddf4a90b0628d178afd67f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:39.620292 14115 raft_consensus.cc:729] T f068a4ffdeb547ea97c691b575b5e0af P 3cf75338fddf4a90b0628d178afd67f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cf75338fddf4a90b0628d178afd67f1, State: Initialized, Role: FOLLOWER
I20260620 18:42:39.620558 14115 consensus_queue.cc:260] T f068a4ffdeb547ea97c691b575b5e0af P 3cf75338fddf4a90b0628d178afd67f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } }
I20260620 18:42:39.623238 14115 ts_tablet_manager.cc:1428] T f068a4ffdeb547ea97c691b575b5e0af P 3cf75338fddf4a90b0628d178afd67f1: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 18:42:39.628978 14111 tablet_bootstrap.cc:492] T f068a4ffdeb547ea97c691b575b5e0af P 06416c21938742de87f05ca67fcc42dc: No bootstrap required, opened a new log
I20260620 18:42:39.629160 14111 ts_tablet_manager.cc:1397] T f068a4ffdeb547ea97c691b575b5e0af P 06416c21938742de87f05ca67fcc42dc: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.008s
I20260620 18:42:39.629961 14111 raft_consensus.cc:348] T f068a4ffdeb547ea97c691b575b5e0af P 06416c21938742de87f05ca67fcc42dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } }
I20260620 18:42:39.630237 14111 raft_consensus.cc:374] T f068a4ffdeb547ea97c691b575b5e0af P 06416c21938742de87f05ca67fcc42dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:39.630340 14111 raft_consensus.cc:729] T f068a4ffdeb547ea97c691b575b5e0af P 06416c21938742de87f05ca67fcc42dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06416c21938742de87f05ca67fcc42dc, State: Initialized, Role: FOLLOWER
I20260620 18:42:39.630641 14111 consensus_queue.cc:260] T f068a4ffdeb547ea97c691b575b5e0af P 06416c21938742de87f05ca67fcc42dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } }
I20260620 18:42:39.631511 14112 tablet_bootstrap.cc:492] T f068a4ffdeb547ea97c691b575b5e0af P 789b964919a043e6a2b7130b340139af: No bootstrap required, opened a new log
I20260620 18:42:39.631664 14112 ts_tablet_manager.cc:1397] T f068a4ffdeb547ea97c691b575b5e0af P 789b964919a043e6a2b7130b340139af: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.006s
I20260620 18:42:39.632197 14111 ts_tablet_manager.cc:1428] T f068a4ffdeb547ea97c691b575b5e0af P 06416c21938742de87f05ca67fcc42dc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 18:42:39.632411 14112 raft_consensus.cc:348] T f068a4ffdeb547ea97c691b575b5e0af P 789b964919a043e6a2b7130b340139af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } }
I20260620 18:42:39.632647 14112 raft_consensus.cc:374] T f068a4ffdeb547ea97c691b575b5e0af P 789b964919a043e6a2b7130b340139af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:39.632726 14112 raft_consensus.cc:729] T f068a4ffdeb547ea97c691b575b5e0af P 789b964919a043e6a2b7130b340139af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789b964919a043e6a2b7130b340139af, State: Initialized, Role: FOLLOWER
I20260620 18:42:39.632938 14112 consensus_queue.cc:260] T f068a4ffdeb547ea97c691b575b5e0af P 789b964919a043e6a2b7130b340139af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } }
I20260620 18:42:39.633559 14112 ts_tablet_manager.cc:1428] T f068a4ffdeb547ea97c691b575b5e0af P 789b964919a043e6a2b7130b340139af: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 18:42:39.785342 14119 raft_consensus.cc:482] T f068a4ffdeb547ea97c691b575b5e0af P 3cf75338fddf4a90b0628d178afd67f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:42:39.785573 14119 raft_consensus.cc:504] T f068a4ffdeb547ea97c691b575b5e0af P 3cf75338fddf4a90b0628d178afd67f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } }
I20260620 18:42:39.787155 14119 leader_election.cc:288] T f068a4ffdeb547ea97c691b575b5e0af P 3cf75338fddf4a90b0628d178afd67f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06416c21938742de87f05ca67fcc42dc (127.5.225.129:44023), 789b964919a043e6a2b7130b340139af (127.5.225.131:37959)
I20260620 18:42:39.795202 14106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:39.787664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:54793 (local address 127.5.225.131:37959)
0620 18:42:39.787949 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:39.787958 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:39.788468 (+ 510us) server_negotiation.cc:407] Connection header received
0620 18:42:39.788618 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:39.788623 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:39.788692 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:39.788807 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:39.789358 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:39.790426 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:39.792802 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:39.794001 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:39.794187 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:39.794255 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:39.794782 (+ 527us) server_negotiation.cc:299] Negotiation successful
0620 18:42:39.794910 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 18:42:39.795186 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:39.787437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:50045 (local address 127.5.225.129:44023)
0620 18:42:39.787718 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:39.787725 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:39.788170 (+ 445us) server_negotiation.cc:407] Connection header received
0620 18:42:39.788398 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:39.788402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:39.788489 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:39.788609 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:39.789074 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:39.790142 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:39.792698 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:39.793998 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:39.794178 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:39.794257 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:39.794782 (+ 525us) server_negotiation.cc:299] Negotiation successful
0620 18:42:39.794913 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:42:39.795383 14191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:39.787371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:44023 (local address 127.5.225.130:50045)
0620 18:42:39.788066 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:39.788084 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:39.788245 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:39.788653 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:39.788657 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:39.788682 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:39.788930 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:39.788947 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:39.790300 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:39.790304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:39.792530 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:39.792546 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:39.794229 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:39.794234 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:39.794530 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:39.794672 (+ 142us) client_negotiation.cc:769] Sending connection context
0620 18:42:39.794750 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 18:42:39.794910 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"mutex_wait_us":168,"thread_start_us":479,"threads_started":1}
I20260620 18:42:39.795408 14192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:39.787412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.131:37959 (local address 127.5.225.130:54793)
0620 18:42:39.788373 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:39.788386 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:39.788485 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:39.788825 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:39.788831 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:39.788860 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:39.789231 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:39.789239 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:39.790538 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:39.790542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:39.792664 (+ 2122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:39.792676 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:39.794195 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:39.794204 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:39.794514 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:39.794611 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 18:42:39.794728 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 18:42:39.794910 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":862,"mutex_wait_us":112,"thread_start_us":685,"threads_started":1}
I20260620 18:42:39.796124 14041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f068a4ffdeb547ea97c691b575b5e0af" candidate_uuid: "3cf75338fddf4a90b0628d178afd67f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789b964919a043e6a2b7130b340139af" is_pre_election: true
I20260620 18:42:39.796123 13763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f068a4ffdeb547ea97c691b575b5e0af" candidate_uuid: "3cf75338fddf4a90b0628d178afd67f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06416c21938742de87f05ca67fcc42dc" is_pre_election: true
I20260620 18:42:39.796368 13763 raft_consensus.cc:2454] T f068a4ffdeb547ea97c691b575b5e0af P 06416c21938742de87f05ca67fcc42dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cf75338fddf4a90b0628d178afd67f1 in term 0.
I20260620 18:42:39.796365 14041 raft_consensus.cc:2454] T f068a4ffdeb547ea97c691b575b5e0af P 789b964919a043e6a2b7130b340139af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cf75338fddf4a90b0628d178afd67f1 in term 0.
I20260620 18:42:39.796864 13831 leader_election.cc:302] T f068a4ffdeb547ea97c691b575b5e0af P 3cf75338fddf4a90b0628d178afd67f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 06416c21938742de87f05ca67fcc42dc, 3cf75338fddf4a90b0628d178afd67f1, 789b964919a043e6a2b7130b340139af; no voters:
I20260620 18:42:39.797276 14119 raft_consensus.cc:2781] T f068a4ffdeb547ea97c691b575b5e0af P 3cf75338fddf4a90b0628d178afd67f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:42:39.797399 14119 raft_consensus.cc:482] T f068a4ffdeb547ea97c691b575b5e0af P 3cf75338fddf4a90b0628d178afd67f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:42:39.797518 14119 raft_consensus.cc:3037] T f068a4ffdeb547ea97c691b575b5e0af P 3cf75338fddf4a90b0628d178afd67f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:39.799911 14119 raft_consensus.cc:504] T f068a4ffdeb547ea97c691b575b5e0af P 3cf75338fddf4a90b0628d178afd67f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } }
I20260620 18:42:39.800622 14119 leader_election.cc:288] T f068a4ffdeb547ea97c691b575b5e0af P 3cf75338fddf4a90b0628d178afd67f1 [CANDIDATE]: Term 1 election: Requested vote from peers 06416c21938742de87f05ca67fcc42dc (127.5.225.129:44023), 789b964919a043e6a2b7130b340139af (127.5.225.131:37959)
I20260620 18:42:39.801098 14041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f068a4ffdeb547ea97c691b575b5e0af" candidate_uuid: "3cf75338fddf4a90b0628d178afd67f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789b964919a043e6a2b7130b340139af"
I20260620 18:42:39.801117 13763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f068a4ffdeb547ea97c691b575b5e0af" candidate_uuid: "3cf75338fddf4a90b0628d178afd67f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06416c21938742de87f05ca67fcc42dc"
I20260620 18:42:39.801338 14041 raft_consensus.cc:3037] T f068a4ffdeb547ea97c691b575b5e0af P 789b964919a043e6a2b7130b340139af [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:39.801349 13763 raft_consensus.cc:3037] T f068a4ffdeb547ea97c691b575b5e0af P 06416c21938742de87f05ca67fcc42dc [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:39.804241 13763 raft_consensus.cc:2454] T f068a4ffdeb547ea97c691b575b5e0af P 06416c21938742de87f05ca67fcc42dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cf75338fddf4a90b0628d178afd67f1 in term 1.
I20260620 18:42:39.804234 14041 raft_consensus.cc:2454] T f068a4ffdeb547ea97c691b575b5e0af P 789b964919a043e6a2b7130b340139af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cf75338fddf4a90b0628d178afd67f1 in term 1.
I20260620 18:42:39.804642 13831 leader_election.cc:302] T f068a4ffdeb547ea97c691b575b5e0af P 3cf75338fddf4a90b0628d178afd67f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06416c21938742de87f05ca67fcc42dc, 3cf75338fddf4a90b0628d178afd67f1; no voters:
I20260620 18:42:39.804935 14119 raft_consensus.cc:2781] T f068a4ffdeb547ea97c691b575b5e0af P 3cf75338fddf4a90b0628d178afd67f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:42:39.805348 14119 raft_consensus.cc:686] T f068a4ffdeb547ea97c691b575b5e0af P 3cf75338fddf4a90b0628d178afd67f1 [term 1 LEADER]: Becoming Leader. State: Replica: 3cf75338fddf4a90b0628d178afd67f1, State: Running, Role: LEADER
I20260620 18:42:39.805708 14119 consensus_queue.cc:237] T f068a4ffdeb547ea97c691b575b5e0af P 3cf75338fddf4a90b0628d178afd67f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } }
I20260620 18:42:39.809384 13599 catalog_manager.cc:5495] T f068a4ffdeb547ea97c691b575b5e0af P 3cf75338fddf4a90b0628d178afd67f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cf75338fddf4a90b0628d178afd67f1 (127.5.225.130). New cstate: current_term: 1 leader_uuid: "3cf75338fddf4a90b0628d178afd67f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20260620 18:42:39.867635 14196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:39.860881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46794 (local address 127.5.225.130:39239)
0620 18:42:39.861534 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:39.861539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:39.861560 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:42:39.861650 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:39.861653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:39.861732 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:39.861921 (+ 189us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:42:39.862460 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:39.863303 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:39.864768 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:39.865241 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:39.865382 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:39.865915 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:42:39.866812 (+ 897us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:42:39.867002 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:39.867070 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 18:42:39.867176 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":352,"threads_started":1}
I20260620 18:42:39.871383 13763 raft_consensus.cc:1261] T f068a4ffdeb547ea97c691b575b5e0af P 06416c21938742de87f05ca67fcc42dc [term 1 FOLLOWER]: Refusing update from remote peer 3cf75338fddf4a90b0628d178afd67f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:42:39.871383 14041 raft_consensus.cc:1261] T f068a4ffdeb547ea97c691b575b5e0af P 789b964919a043e6a2b7130b340139af [term 1 FOLLOWER]: Refusing update from remote peer 3cf75338fddf4a90b0628d178afd67f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:42:39.872112 14193 consensus_queue.cc:1035] T f068a4ffdeb547ea97c691b575b5e0af P 3cf75338fddf4a90b0628d178afd67f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:42:39.872517 14119 consensus_queue.cc:1035] T f068a4ffdeb547ea97c691b575b5e0af P 3cf75338fddf4a90b0628d178afd67f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:42:39.888818 13600 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:42:39.889056 13600 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:42:39.892023 13600 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 972f20065ea9422a838127c325ed3908: Sending DeleteTablet for 3 replicas of tablet f068a4ffdeb547ea97c691b575b5e0af
I20260620 18:42:39.892771 13738 tablet_service.cc:1514] Processing DeleteTablet for tablet f068a4ffdeb547ea97c691b575b5e0af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:42:39 UTC) from {username='oryx', principal='oryx/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:57858
I20260620 18:42:39.892834 13877 tablet_service.cc:1514] Processing DeleteTablet for tablet f068a4ffdeb547ea97c691b575b5e0af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:42:39 UTC) from {username='oryx', principal='oryx/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:46782
I20260620 18:42:39.893085 14173 tablet_replica.cc:330] T f068a4ffdeb547ea97c691b575b5e0af P 06416c21938742de87f05ca67fcc42dc: stopping tablet replica
I20260620 18:42:39.893155 14020 tablet_service.cc:1514] Processing DeleteTablet for tablet f068a4ffdeb547ea97c691b575b5e0af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:42:39 UTC) from {username='oryx', principal='oryx/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:57452
I20260620 18:42:39.893291 14173 raft_consensus.cc:2229] T f068a4ffdeb547ea97c691b575b5e0af P 06416c21938742de87f05ca67fcc42dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:42:39.893414 14174 tablet_replica.cc:330] T f068a4ffdeb547ea97c691b575b5e0af P 3cf75338fddf4a90b0628d178afd67f1: stopping tablet replica
I20260620 18:42:39.893513 14173 raft_consensus.cc:2258] T f068a4ffdeb547ea97c691b575b5e0af P 06416c21938742de87f05ca67fcc42dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:42:39.893610 14174 raft_consensus.cc:2229] T f068a4ffdeb547ea97c691b575b5e0af P 3cf75338fddf4a90b0628d178afd67f1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:42:39.894121 14174 raft_consensus.cc:2258] T f068a4ffdeb547ea97c691b575b5e0af P 3cf75338fddf4a90b0628d178afd67f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:42:39.894402 14175 tablet_replica.cc:330] T f068a4ffdeb547ea97c691b575b5e0af P 789b964919a043e6a2b7130b340139af: stopping tablet replica
I20260620 18:42:39.894585 14175 raft_consensus.cc:2229] T f068a4ffdeb547ea97c691b575b5e0af P 789b964919a043e6a2b7130b340139af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:42:39.894369 13600 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 18:42:39.894786 14175 raft_consensus.cc:2258] T f068a4ffdeb547ea97c691b575b5e0af P 789b964919a043e6a2b7130b340139af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:42:39.895013 14173 ts_tablet_manager.cc:1905] T f068a4ffdeb547ea97c691b575b5e0af P 06416c21938742de87f05ca67fcc42dc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 18:42:39.895080 13600 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:42:39.896289 14174 ts_tablet_manager.cc:1905] T f068a4ffdeb547ea97c691b575b5e0af P 3cf75338fddf4a90b0628d178afd67f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:42:39.902230 13738 tablet_service.cc:1467] Processing CreateTablet for tablet 2d2d5aa22fc94024a8ed77e59513be4f (DEFAULT_TABLE table=test-table [id=efbc9be3a9fd453f85846d76a031b092]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:42:39.902678 13738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d2d5aa22fc94024a8ed77e59513be4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:39.902832 14174 ts_tablet_manager.cc:1918] T f068a4ffdeb547ea97c691b575b5e0af P 3cf75338fddf4a90b0628d178afd67f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:42:39.902987 14174 log.cc:1196] T f068a4ffdeb547ea97c691b575b5e0af P 3cf75338fddf4a90b0628d178afd67f1: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-1/wal/wals/f068a4ffdeb547ea97c691b575b5e0af
I20260620 18:42:39.903326 13877 tablet_service.cc:1467] Processing CreateTablet for tablet 2d2d5aa22fc94024a8ed77e59513be4f (DEFAULT_TABLE table=test-table [id=efbc9be3a9fd453f85846d76a031b092]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:42:39.903475 14174 ts_tablet_manager.cc:1939] T f068a4ffdeb547ea97c691b575b5e0af P 3cf75338fddf4a90b0628d178afd67f1: Deleting consensus metadata
I20260620 18:42:39.903755 13877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d2d5aa22fc94024a8ed77e59513be4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:39.904973 14175 ts_tablet_manager.cc:1905] T f068a4ffdeb547ea97c691b575b5e0af P 789b964919a043e6a2b7130b340139af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:42:39.907207 14020 tablet_service.cc:1467] Processing CreateTablet for tablet 2d2d5aa22fc94024a8ed77e59513be4f (DEFAULT_TABLE table=test-table [id=efbc9be3a9fd453f85846d76a031b092]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:42:39.907653 14020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d2d5aa22fc94024a8ed77e59513be4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:39.909626 14111 tablet_bootstrap.cc:492] T 2d2d5aa22fc94024a8ed77e59513be4f P 06416c21938742de87f05ca67fcc42dc: Bootstrap starting.
I20260620 18:42:39.910286 13585 catalog_manager.cc:4841] TS 3cf75338fddf4a90b0628d178afd67f1 (127.5.225.130:39239): tablet f068a4ffdeb547ea97c691b575b5e0af (table test-table [id=c36b4f5c11cc43aab31246e6619620f3]) successfully deleted
I20260620 18:42:39.911281 14111 tablet_bootstrap.cc:654] T 2d2d5aa22fc94024a8ed77e59513be4f P 06416c21938742de87f05ca67fcc42dc: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:39.911618 14175 ts_tablet_manager.cc:1918] T f068a4ffdeb547ea97c691b575b5e0af P 789b964919a043e6a2b7130b340139af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:42:39.911762 14175 log.cc:1196] T f068a4ffdeb547ea97c691b575b5e0af P 789b964919a043e6a2b7130b340139af: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-2/wal/wals/f068a4ffdeb547ea97c691b575b5e0af
I20260620 18:42:39.912243 14175 ts_tablet_manager.cc:1939] T f068a4ffdeb547ea97c691b575b5e0af P 789b964919a043e6a2b7130b340139af: Deleting consensus metadata
I20260620 18:42:39.902685 14173 ts_tablet_manager.cc:1918] T f068a4ffdeb547ea97c691b575b5e0af P 06416c21938742de87f05ca67fcc42dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:42:39.915638 14173 log.cc:1196] T f068a4ffdeb547ea97c691b575b5e0af P 06416c21938742de87f05ca67fcc42dc: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-0/wal/wals/f068a4ffdeb547ea97c691b575b5e0af
I20260620 18:42:39.916064 14173 ts_tablet_manager.cc:1939] T f068a4ffdeb547ea97c691b575b5e0af P 06416c21938742de87f05ca67fcc42dc: Deleting consensus metadata
I20260620 18:42:39.917053 13588 catalog_manager.cc:4841] TS 06416c21938742de87f05ca67fcc42dc (127.5.225.129:44023): tablet f068a4ffdeb547ea97c691b575b5e0af (table test-table [id=c36b4f5c11cc43aab31246e6619620f3]) successfully deleted
I20260620 18:42:39.918016 14112 tablet_bootstrap.cc:492] T 2d2d5aa22fc94024a8ed77e59513be4f P 789b964919a043e6a2b7130b340139af: Bootstrap starting.
I20260620 18:42:39.920089 14112 tablet_bootstrap.cc:654] T 2d2d5aa22fc94024a8ed77e59513be4f P 789b964919a043e6a2b7130b340139af: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:39.920964 14115 tablet_bootstrap.cc:492] T 2d2d5aa22fc94024a8ed77e59513be4f P 3cf75338fddf4a90b0628d178afd67f1: Bootstrap starting.
I20260620 18:42:39.922518 14111 tablet_bootstrap.cc:492] T 2d2d5aa22fc94024a8ed77e59513be4f P 06416c21938742de87f05ca67fcc42dc: No bootstrap required, opened a new log
I20260620 18:42:39.922811 14111 ts_tablet_manager.cc:1397] T 2d2d5aa22fc94024a8ed77e59513be4f P 06416c21938742de87f05ca67fcc42dc: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.009s
I20260620 18:42:39.922863 14115 tablet_bootstrap.cc:654] T 2d2d5aa22fc94024a8ed77e59513be4f P 3cf75338fddf4a90b0628d178afd67f1: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:39.924983 14111 raft_consensus.cc:348] T 2d2d5aa22fc94024a8ed77e59513be4f P 06416c21938742de87f05ca67fcc42dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } }
I20260620 18:42:39.925359 14111 raft_consensus.cc:374] T 2d2d5aa22fc94024a8ed77e59513be4f P 06416c21938742de87f05ca67fcc42dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:39.925455 14111 raft_consensus.cc:729] T 2d2d5aa22fc94024a8ed77e59513be4f P 06416c21938742de87f05ca67fcc42dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06416c21938742de87f05ca67fcc42dc, State: Initialized, Role: FOLLOWER
I20260620 18:42:39.925812 14111 consensus_queue.cc:260] T 2d2d5aa22fc94024a8ed77e59513be4f P 06416c21938742de87f05ca67fcc42dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } }
I20260620 18:42:39.929558 13586 catalog_manager.cc:4841] TS 789b964919a043e6a2b7130b340139af (127.5.225.131:37959): tablet f068a4ffdeb547ea97c691b575b5e0af (table test-table [id=c36b4f5c11cc43aab31246e6619620f3]) successfully deleted
I20260620 18:42:39.931332 14111 ts_tablet_manager.cc:1428] T 2d2d5aa22fc94024a8ed77e59513be4f P 06416c21938742de87f05ca67fcc42dc: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20260620 18:42:39.935555 14112 tablet_bootstrap.cc:492] T 2d2d5aa22fc94024a8ed77e59513be4f P 789b964919a043e6a2b7130b340139af: No bootstrap required, opened a new log
I20260620 18:42:39.935750 14112 ts_tablet_manager.cc:1397] T 2d2d5aa22fc94024a8ed77e59513be4f P 789b964919a043e6a2b7130b340139af: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.010s
I20260620 18:42:39.936400 14112 raft_consensus.cc:348] T 2d2d5aa22fc94024a8ed77e59513be4f P 789b964919a043e6a2b7130b340139af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } }
I20260620 18:42:39.936537 14115 tablet_bootstrap.cc:492] T 2d2d5aa22fc94024a8ed77e59513be4f P 3cf75338fddf4a90b0628d178afd67f1: No bootstrap required, opened a new log
I20260620 18:42:39.936626 14112 raft_consensus.cc:374] T 2d2d5aa22fc94024a8ed77e59513be4f P 789b964919a043e6a2b7130b340139af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:39.936702 14112 raft_consensus.cc:729] T 2d2d5aa22fc94024a8ed77e59513be4f P 789b964919a043e6a2b7130b340139af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789b964919a043e6a2b7130b340139af, State: Initialized, Role: FOLLOWER
I20260620 18:42:39.936858 14115 ts_tablet_manager.cc:1397] T 2d2d5aa22fc94024a8ed77e59513be4f P 3cf75338fddf4a90b0628d178afd67f1: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.004s
I20260620 18:42:39.936957 14112 consensus_queue.cc:260] T 2d2d5aa22fc94024a8ed77e59513be4f P 789b964919a043e6a2b7130b340139af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } }
I20260620 18:42:39.937498 14112 ts_tablet_manager.cc:1428] T 2d2d5aa22fc94024a8ed77e59513be4f P 789b964919a043e6a2b7130b340139af: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 18:42:39.937723 14115 raft_consensus.cc:348] T 2d2d5aa22fc94024a8ed77e59513be4f P 3cf75338fddf4a90b0628d178afd67f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } }
I20260620 18:42:39.938016 14115 raft_consensus.cc:374] T 2d2d5aa22fc94024a8ed77e59513be4f P 3cf75338fddf4a90b0628d178afd67f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:39.938148 14115 raft_consensus.cc:729] T 2d2d5aa22fc94024a8ed77e59513be4f P 3cf75338fddf4a90b0628d178afd67f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cf75338fddf4a90b0628d178afd67f1, State: Initialized, Role: FOLLOWER
I20260620 18:42:39.938639 14115 consensus_queue.cc:260] T 2d2d5aa22fc94024a8ed77e59513be4f P 3cf75338fddf4a90b0628d178afd67f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } }
I20260620 18:42:39.939229 14115 ts_tablet_manager.cc:1428] T 2d2d5aa22fc94024a8ed77e59513be4f P 3cf75338fddf4a90b0628d178afd67f1: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260620 18:42:40.089991 14171 raft_consensus.cc:482] T 2d2d5aa22fc94024a8ed77e59513be4f P 789b964919a043e6a2b7130b340139af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:42:40.090236 14171 raft_consensus.cc:504] T 2d2d5aa22fc94024a8ed77e59513be4f P 789b964919a043e6a2b7130b340139af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } }
I20260620 18:42:40.090988 14171 leader_election.cc:288] T 2d2d5aa22fc94024a8ed77e59513be4f P 789b964919a043e6a2b7130b340139af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06416c21938742de87f05ca67fcc42dc (127.5.225.129:44023), 3cf75338fddf4a90b0628d178afd67f1 (127.5.225.130:39239)
I20260620 18:42:40.091605 13763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d2d5aa22fc94024a8ed77e59513be4f" candidate_uuid: "789b964919a043e6a2b7130b340139af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06416c21938742de87f05ca67fcc42dc" is_pre_election: true
I20260620 18:42:40.091646 13901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d2d5aa22fc94024a8ed77e59513be4f" candidate_uuid: "789b964919a043e6a2b7130b340139af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cf75338fddf4a90b0628d178afd67f1" is_pre_election: true
I20260620 18:42:40.091874 13763 raft_consensus.cc:2454] T 2d2d5aa22fc94024a8ed77e59513be4f P 06416c21938742de87f05ca67fcc42dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 789b964919a043e6a2b7130b340139af in term 0.
I20260620 18:42:40.091953 13901 raft_consensus.cc:2454] T 2d2d5aa22fc94024a8ed77e59513be4f P 3cf75338fddf4a90b0628d178afd67f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 789b964919a043e6a2b7130b340139af in term 0.
I20260620 18:42:40.092388 13970 leader_election.cc:302] T 2d2d5aa22fc94024a8ed77e59513be4f P 789b964919a043e6a2b7130b340139af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06416c21938742de87f05ca67fcc42dc, 789b964919a043e6a2b7130b340139af; no voters:
I20260620 18:42:40.092692 14171 raft_consensus.cc:2781] T 2d2d5aa22fc94024a8ed77e59513be4f P 789b964919a043e6a2b7130b340139af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:42:40.092816 14171 raft_consensus.cc:482] T 2d2d5aa22fc94024a8ed77e59513be4f P 789b964919a043e6a2b7130b340139af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:42:40.092934 14171 raft_consensus.cc:3037] T 2d2d5aa22fc94024a8ed77e59513be4f P 789b964919a043e6a2b7130b340139af [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:40.094988 14171 raft_consensus.cc:504] T 2d2d5aa22fc94024a8ed77e59513be4f P 789b964919a043e6a2b7130b340139af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } }
I20260620 18:42:40.095556 14171 leader_election.cc:288] T 2d2d5aa22fc94024a8ed77e59513be4f P 789b964919a043e6a2b7130b340139af [CANDIDATE]: Term 1 election: Requested vote from peers 06416c21938742de87f05ca67fcc42dc (127.5.225.129:44023), 3cf75338fddf4a90b0628d178afd67f1 (127.5.225.130:39239)
I20260620 18:42:40.096037 13763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d2d5aa22fc94024a8ed77e59513be4f" candidate_uuid: "789b964919a043e6a2b7130b340139af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06416c21938742de87f05ca67fcc42dc"
I20260620 18:42:40.096100 13901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d2d5aa22fc94024a8ed77e59513be4f" candidate_uuid: "789b964919a043e6a2b7130b340139af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cf75338fddf4a90b0628d178afd67f1"
I20260620 18:42:40.096256 13763 raft_consensus.cc:3037] T 2d2d5aa22fc94024a8ed77e59513be4f P 06416c21938742de87f05ca67fcc42dc [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:40.096338 13901 raft_consensus.cc:3037] T 2d2d5aa22fc94024a8ed77e59513be4f P 3cf75338fddf4a90b0628d178afd67f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:40.098258 13763 raft_consensus.cc:2454] T 2d2d5aa22fc94024a8ed77e59513be4f P 06416c21938742de87f05ca67fcc42dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 789b964919a043e6a2b7130b340139af in term 1.
I20260620 18:42:40.098685 13970 leader_election.cc:302] T 2d2d5aa22fc94024a8ed77e59513be4f P 789b964919a043e6a2b7130b340139af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06416c21938742de87f05ca67fcc42dc, 789b964919a043e6a2b7130b340139af; no voters:
I20260620 18:42:40.098986 14171 raft_consensus.cc:2781] T 2d2d5aa22fc94024a8ed77e59513be4f P 789b964919a043e6a2b7130b340139af [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:42:40.099457 14171 raft_consensus.cc:686] T 2d2d5aa22fc94024a8ed77e59513be4f P 789b964919a043e6a2b7130b340139af [term 1 LEADER]: Becoming Leader. State: Replica: 789b964919a043e6a2b7130b340139af, State: Running, Role: LEADER
I20260620 18:42:40.099638 13901 raft_consensus.cc:2454] T 2d2d5aa22fc94024a8ed77e59513be4f P 3cf75338fddf4a90b0628d178afd67f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 789b964919a043e6a2b7130b340139af in term 1.
I20260620 18:42:40.100164 14171 consensus_queue.cc:237] T 2d2d5aa22fc94024a8ed77e59513be4f P 789b964919a043e6a2b7130b340139af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } }
I20260620 18:42:40.102808 13600 catalog_manager.cc:5495] T 2d2d5aa22fc94024a8ed77e59513be4f P 789b964919a043e6a2b7130b340139af reported cstate change: term changed from 0 to 1, leader changed from <none> to 789b964919a043e6a2b7130b340139af (127.5.225.131). New cstate: current_term: 1 leader_uuid: "789b964919a043e6a2b7130b340139af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "789b964919a043e6a2b7130b340139af" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 37959 } health_report { overall_health: HEALTHY } } }
I20260620 18:42:40.131181 14106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:40.126447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57476 (local address 127.5.225.131:37959)
0620 18:42:40.126598 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:40.126604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:40.126762 (+ 158us) server_negotiation.cc:407] Connection header received
0620 18:42:40.126967 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:40.126972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:40.127046 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:40.127144 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:42:40.127520 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:40.128241 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:40.129499 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:40.129905 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:40.130034 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:40.130501 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:42:40.130606 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:42:40.130700 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:40.130843 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 18:42:40.130917 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:42:40.139678 13763 raft_consensus.cc:1261] T 2d2d5aa22fc94024a8ed77e59513be4f P 06416c21938742de87f05ca67fcc42dc [term 1 FOLLOWER]: Refusing update from remote peer 789b964919a043e6a2b7130b340139af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:42:40.140193 14204 consensus_queue.cc:1035] T 2d2d5aa22fc94024a8ed77e59513be4f P 789b964919a043e6a2b7130b340139af [LEADER]: Connected to new peer: Peer: permanent_uuid: "06416c21938742de87f05ca67fcc42dc" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 44023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:42:40.141115 13901 raft_consensus.cc:1261] T 2d2d5aa22fc94024a8ed77e59513be4f P 3cf75338fddf4a90b0628d178afd67f1 [term 1 FOLLOWER]: Refusing update from remote peer 789b964919a043e6a2b7130b340139af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:42:40.141616 14204 consensus_queue.cc:1035] T 2d2d5aa22fc94024a8ed77e59513be4f P 789b964919a043e6a2b7130b340139af [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cf75338fddf4a90b0628d178afd67f1" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:42:40.193720 13600 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:42:40.193990 13600 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:42:40.197062 13600 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 972f20065ea9422a838127c325ed3908: Sending DeleteTablet for 3 replicas of tablet 2d2d5aa22fc94024a8ed77e59513be4f
I20260620 18:42:40.197775 13738 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d2d5aa22fc94024a8ed77e59513be4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:42:40 UTC) from {username='oryx', principal='oryx/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:57858
I20260620 18:42:40.197875 13877 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d2d5aa22fc94024a8ed77e59513be4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:42:40 UTC) from {username='oryx', principal='oryx/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:46782
I20260620 18:42:40.198067 14173 tablet_replica.cc:330] T 2d2d5aa22fc94024a8ed77e59513be4f P 06416c21938742de87f05ca67fcc42dc: stopping tablet replica
I20260620 18:42:40.198077 14020 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d2d5aa22fc94024a8ed77e59513be4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:42:40 UTC) from {username='oryx', principal='oryx/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:57452
I20260620 18:42:40.198184 14174 tablet_replica.cc:330] T 2d2d5aa22fc94024a8ed77e59513be4f P 3cf75338fddf4a90b0628d178afd67f1: stopping tablet replica
I20260620 18:42:40.198292 14173 raft_consensus.cc:2229] T 2d2d5aa22fc94024a8ed77e59513be4f P 06416c21938742de87f05ca67fcc42dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:42:40.198383 14175 tablet_replica.cc:330] T 2d2d5aa22fc94024a8ed77e59513be4f P 789b964919a043e6a2b7130b340139af: stopping tablet replica
I20260620 18:42:40.198377 14174 raft_consensus.cc:2229] T 2d2d5aa22fc94024a8ed77e59513be4f P 3cf75338fddf4a90b0628d178afd67f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:42:40.198498 14173 raft_consensus.cc:2258] T 2d2d5aa22fc94024a8ed77e59513be4f P 06416c21938742de87f05ca67fcc42dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:42:40.198534 14175 raft_consensus.cc:2229] T 2d2d5aa22fc94024a8ed77e59513be4f P 789b964919a043e6a2b7130b340139af [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:42:40.198575 14174 raft_consensus.cc:2258] T 2d2d5aa22fc94024a8ed77e59513be4f P 3cf75338fddf4a90b0628d178afd67f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:42:40.198868 14175 raft_consensus.cc:2258] T 2d2d5aa22fc94024a8ed77e59513be4f P 789b964919a043e6a2b7130b340139af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:42:40.199684 14173 ts_tablet_manager.cc:1905] T 2d2d5aa22fc94024a8ed77e59513be4f P 06416c21938742de87f05ca67fcc42dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:42:40.200040 14175 ts_tablet_manager.cc:1905] T 2d2d5aa22fc94024a8ed77e59513be4f P 789b964919a043e6a2b7130b340139af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:42:40.200481 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 13672
I20260620 18:42:40.207262 14173 ts_tablet_manager.cc:1918] T 2d2d5aa22fc94024a8ed77e59513be4f P 06416c21938742de87f05ca67fcc42dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:42:40.207406 14173 log.cc:1196] T 2d2d5aa22fc94024a8ed77e59513be4f P 06416c21938742de87f05ca67fcc42dc: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-0/wal/wals/2d2d5aa22fc94024a8ed77e59513be4f
I20260620 18:42:40.207733 14174 ts_tablet_manager.cc:1905] T 2d2d5aa22fc94024a8ed77e59513be4f P 3cf75338fddf4a90b0628d178afd67f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:42:40.207854 14173 ts_tablet_manager.cc:1939] T 2d2d5aa22fc94024a8ed77e59513be4f P 06416c21938742de87f05ca67fcc42dc: Deleting consensus metadata
I20260620 18:42:40.208957 13588 catalog_manager.cc:4841] TS 06416c21938742de87f05ca67fcc42dc (127.5.225.129:44023): tablet 2d2d5aa22fc94024a8ed77e59513be4f (table test-table [id=efbc9be3a9fd453f85846d76a031b092]) successfully deleted
I20260620 18:42:40.212031 14175 ts_tablet_manager.cc:1918] T 2d2d5aa22fc94024a8ed77e59513be4f P 789b964919a043e6a2b7130b340139af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:42:40.212231 14175 log.cc:1196] T 2d2d5aa22fc94024a8ed77e59513be4f P 789b964919a043e6a2b7130b340139af: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-2/wal/wals/2d2d5aa22fc94024a8ed77e59513be4f
I20260620 18:42:40.212689 14175 ts_tablet_manager.cc:1939] T 2d2d5aa22fc94024a8ed77e59513be4f P 789b964919a043e6a2b7130b340139af: Deleting consensus metadata
I20260620 18:42:40.213789 13586 catalog_manager.cc:4841] TS 789b964919a043e6a2b7130b340139af (127.5.225.131:37959): tablet 2d2d5aa22fc94024a8ed77e59513be4f (table test-table [id=efbc9be3a9fd453f85846d76a031b092]) successfully deleted
I20260620 18:42:40.215801 14174 ts_tablet_manager.cc:1918] T 2d2d5aa22fc94024a8ed77e59513be4f P 3cf75338fddf4a90b0628d178afd67f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:42:40.215948 14174 log.cc:1196] T 2d2d5aa22fc94024a8ed77e59513be4f P 3cf75338fddf4a90b0628d178afd67f1: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980894474392-6022-0/minicluster-data/ts-1/wal/wals/2d2d5aa22fc94024a8ed77e59513be4f
I20260620 18:42:40.216362 14174 ts_tablet_manager.cc:1939] T 2d2d5aa22fc94024a8ed77e59513be4f P 3cf75338fddf4a90b0628d178afd67f1: Deleting consensus metadata
I20260620 18:42:40.217444 13585 catalog_manager.cc:4841] TS 3cf75338fddf4a90b0628d178afd67f1 (127.5.225.130:39239): tablet 2d2d5aa22fc94024a8ed77e59513be4f (table test-table [id=efbc9be3a9fd453f85846d76a031b092]) successfully deleted
Jun 20 18:42:40 dist-test-slave-k6lg krb5kdc[13556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.225.129@KRBTEST.COM
I20260620 18:42:40.231462 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:40.201772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57884 (local address 127.5.225.129:44023)
0620 18:42:40.202404 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:40.202411 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:40.214557 (+ 12146us) server_negotiation.cc:407] Connection header received
0620 18:42:40.214748 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:40.214754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:40.214836 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:40.214942 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:40.216564 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:40.217631 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:40.219490 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:40.219892 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:40.219983 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:40.224657 (+ 4674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:40.224675 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:40.224695 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:40.224740 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:40.228503 (+ 3763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:40.229268 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:40.229278 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:40.229282 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:40.229436 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:40.229812 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:40.229817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:40.229821 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:40.230369 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:40.230662 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:40.230858 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 18:42:40.231081 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505}
I20260620 18:42:40.232084 13803 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:40.354285 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 13672
I20260620 18:42:40.377218 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 13812
Jun 20 18:42:40 dist-test-slave-k6lg krb5kdc[13556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.225.130@KRBTEST.COM
I20260620 18:42:40.392375 14215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:40.377965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46810 (local address 127.5.225.130:39239)
0620 18:42:40.378488 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:40.378496 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:40.378518 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:42:40.378606 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:40.378615 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:40.378719 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:40.378837 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:40.380027 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:40.380719 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:40.381972 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:40.382545 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:40.382673 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:40.386925 (+ 4252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:40.386941 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:40.386965 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:40.387003 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:40.389567 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:40.390276 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:40.390286 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:40.390290 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:40.390427 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:40.390794 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:40.390799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:40.390803 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:40.391224 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:40.391524 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:40.391817 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 18:42:40.392030 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":260,"threads_started":1}
I20260620 18:42:40.393115 13942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:40.515271 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 13812
I20260620 18:42:40.538079 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 13951
Jun 20 18:42:40 dist-test-slave-k6lg krb5kdc[13556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.225.131@KRBTEST.COM
I20260620 18:42:40.553037 14106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:40.538862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57490 (local address 127.5.225.131:37959)
0620 18:42:40.539044 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:40.539049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:40.539068 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:42:40.539204 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:40.539208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:40.539282 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:40.539381 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:40.540585 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:40.541321 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:40.542803 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:40.543223 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:40.543341 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:40.547605 (+ 4264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:40.547621 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:40.547645 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:40.547681 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:40.550438 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:40.551100 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:40.551108 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:40.551111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:40.551261 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:40.551623 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:40.551628 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:40.551630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:40.552055 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:40.552336 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:40.552522 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 18:42:40.552761 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:42:40.553833 14081 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:40.675710 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 13951
I20260620 18:42:40.700453 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 13569
I20260620 18:42:40.701607 13641 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:40.817107 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 13569
2026-06-20T18:42:40Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6693 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:42:40 dist-test-slave-k6lg krb5kdc[14221](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:42:40 dist-test-slave-k6lg krb5kdc[14221](info): set up 2 sockets
Jun 20 18:42:40 dist-test-slave-k6lg krb5kdc[14221](info): commencing operation
krb5kdc: starting...
W20260620 18:42:41.911350 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.003s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:42:42 dist-test-slave-k6lg krb5kdc[14221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980962, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:42:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:42:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
I20260620 18:42:42.123726 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:40581
--webserver_interface=127.5.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:39989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:40581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:42.446882 14234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:42.447170 14234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:42.447261 14234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:42.454332 14234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:42:42.454437 14234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:42.454484 14234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:42.454536 14234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:42:42.454576 14234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:42:42.464243 14234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:39989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:40581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:40581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:42.465456 14234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:42.466712 14234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:42.478470 14239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:42.478718 14240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:42.478686 14242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:42.479547 14234 server_base.cc:1034] running on GCE node
I20260620 18:42:42.480330 14234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:42.481796 14234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:42.483001 14234 hybrid_clock.cc:648] HybridClock initialized: now 1781980962482939 us; error 52 us; skew 500 ppm
I20260620 18:42:42.483320 14234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T18:42:42Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1730 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:42:42 dist-test-slave-k6lg krb5kdc[14248](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:42:42 dist-test-slave-k6lg krb5kdc[14248](info): set up 2 sockets
Jun 20 18:42:42 dist-test-slave-k6lg krb5kdc[14248](info): commencing operation
krb5kdc: starting...
W20260620 18:42:43.647190 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.005s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:42:43 dist-test-slave-k6lg krb5kdc[14248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:42:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:42:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
I20260620 18:42:43.862090 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:39687
--webserver_interface=127.5.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:37003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:39687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:44.202829 14261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:44.203099 14261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:44.203192 14261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:44.210090 14261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:42:44.210239 14261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:44.210309 14261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:44.210371 14261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:42:44.210408 14261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:42:44.220103 14261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:37003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:39687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:39687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:44.221379 14261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:44.222667 14261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:44.234593 14266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:44.234503 14267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:44.234539 14269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:44.235657 14261 server_base.cc:1034] running on GCE node
I20260620 18:42:44.236429 14261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:44.237855 14261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:44.239099 14261 hybrid_clock.cc:648] HybridClock initialized: now 1781980964239053 us; error 57 us; skew 500 ppm
I20260620 18:42:44.239462 14261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:44 dist-test-slave-k6lg krb5kdc[14248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980964, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:44.242769 14261 init.cc:377] Logged in from keytab as kudu/127.5.225.190@KRBTEST.COM (short username kudu)
I20260620 18:42:44.244423 14261 webserver.cc:460] Webserver started at http://127.5.225.190:39053/ using document root <none> and password file <none>
I20260620 18:42:44.245169 14261 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:44.245319 14261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:44.245656 14261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:44.248730 14261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "8a3de271691d4d81b5faa32a51a8fae0"
format_stamp: "Formatted at 2026-06-20 18:42:44 on dist-test-slave-k6lg"
server_key: "e53c3d8a92e4ccac898436bca708cd18"
server_key_iv: "d1fa8cde9f23dc1ad867516912d167a1"
server_key_version: "encryptionkey@0"
I20260620 18:42:44.249498 14261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "8a3de271691d4d81b5faa32a51a8fae0"
format_stamp: "Formatted at 2026-06-20 18:42:44 on dist-test-slave-k6lg"
server_key: "e53c3d8a92e4ccac898436bca708cd18"
server_key_iv: "d1fa8cde9f23dc1ad867516912d167a1"
server_key_version: "encryptionkey@0"
I20260620 18:42:44.254596 14261 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 18:42:44.257889 14276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:44.259305 14261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:42:44.259498 14261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "8a3de271691d4d81b5faa32a51a8fae0"
format_stamp: "Formatted at 2026-06-20 18:42:44 on dist-test-slave-k6lg"
server_key: "e53c3d8a92e4ccac898436bca708cd18"
server_key_iv: "d1fa8cde9f23dc1ad867516912d167a1"
server_key_version: "encryptionkey@0"
I20260620 18:42:44.259692 14261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:44.271198 14261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:44.275466 14261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:44.275749 14261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:44.293584 14261 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:39687
I20260620 18:42:44.293606 14327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:39687 every 8 connection(s)
I20260620 18:42:44.295022 14261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:42:44.296744 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 14261
I20260620 18:42:44.296950 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:42:44.297731 6022 external_mini_cluster.cc:1428] Setting key cf1617a0b8cee686a3ae1c968d22e732
I20260620 18:42:44.299504 14328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 18:42:44 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:42:44.312927 14328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0: Bootstrap starting.
I20260620 18:42:44.316054 14328 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:44.317016 14328 log.cc:824] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:44.320003 14328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0: No bootstrap required, opened a new log
I20260620 18:42:44.320109 14331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:44.299718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60528 (local address 127.5.225.190:39687)
0620 18:42:44.300474 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:44.300486 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:44.300523 (+ 37us) server_negotiation.cc:407] Connection header received
0620 18:42:44.302045 (+ 1522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:44.302065 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:44.302492 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:44.302811 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:44.304592 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:44.305891 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:44.307713 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:44.308369 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:44.308500 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:44.312962 (+ 4462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:44.312969 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:44.312983 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:44.313049 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:44.316131 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:44.316766 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:44.316777 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:44.316789 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:44.316943 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:44.317351 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:44.317356 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:44.317358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:44.317832 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:44.318143 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:44.318573 (+ 430us) server_negotiation.cc:299] Negotiation successful
0620 18:42:44.318978 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":336,"threads_started":1}
I20260620 18:42:44.328567 14328 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a3de271691d4d81b5faa32a51a8fae0" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39687 } }
I20260620 18:42:44.328972 14328 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:44.329106 14328 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a3de271691d4d81b5faa32a51a8fae0, State: Initialized, Role: FOLLOWER
I20260620 18:42:44.329656 14328 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a3de271691d4d81b5faa32a51a8fae0" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39687 } }
I20260620 18:42:44.329890 14328 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:42:44.330037 14328 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:42:44.330221 14328 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:44.332283 14328 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a3de271691d4d81b5faa32a51a8fae0" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39687 } }
I20260620 18:42:44.332799 14328 leader_election.cc:302] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a3de271691d4d81b5faa32a51a8fae0; no voters:
I20260620 18:42:44.333343 14328 leader_election.cc:288] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:42:44.333485 14333 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:42:44.334208 14333 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0 [term 1 LEADER]: Becoming Leader. State: Replica: 8a3de271691d4d81b5faa32a51a8fae0, State: Running, Role: LEADER
I20260620 18:42:44.334709 14333 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a3de271691d4d81b5faa32a51a8fae0" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39687 } }
I20260620 18:42:44.335467 14328 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:42:44.337225 14335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a3de271691d4d81b5faa32a51a8fae0. Latest consensus state: current_term: 1 leader_uuid: "8a3de271691d4d81b5faa32a51a8fae0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a3de271691d4d81b5faa32a51a8fae0" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39687 } } }
I20260620 18:42:44.337738 14335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0 [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:44.337738 14334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a3de271691d4d81b5faa32a51a8fae0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a3de271691d4d81b5faa32a51a8fae0" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39687 } } }
I20260620 18:42:44.338056 14334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0 [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:44.339570 14338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:42:44.344312 14338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:42:44.353609 14338 catalog_manager.cc:1347] Generated new cluster ID: 363a991473af4483a4cd9ff2b3e479e3
I20260620 18:42:44.353717 14338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:42:44.391886 14338 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:42:44.393769 14338 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:42:44.404881 14338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0: Generated new TSK 0
I20260620 18:42:44.405926 14338 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab.
Entry for principal kudu/127.5.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab.
I20260620 18:42:44.461186 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.129:0
--local_ip_for_outbound_sockets=127.5.225.129
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:39687
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:37003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:44.780823 14354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:44.781101 14354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:44.781279 14354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:44.789016 14354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:44.789140 14354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:44.789319 14354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.129
I20260620 18:42:44.798801 14354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:37003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:39687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.129
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:44.800064 14354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:44.801359 14354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:44.814702 14359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:44.814738 14360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:44.814778 14362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:44.815250 14354 server_base.cc:1034] running on GCE node
I20260620 18:42:44.815907 14354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:44.816880 14354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:44.818094 14354 hybrid_clock.cc:648] HybridClock initialized: now 1781980964818056 us; error 34 us; skew 500 ppm
I20260620 18:42:44.818452 14354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:44 dist-test-slave-k6lg krb5kdc[14248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980964, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:44.821617 14354 init.cc:377] Logged in from keytab as kudu/127.5.225.129@KRBTEST.COM (short username kudu)
I20260620 18:42:44.823078 14354 webserver.cc:460] Webserver started at http://127.5.225.129:36775/ using document root <none> and password file <none>
I20260620 18:42:44.823778 14354 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:44.823922 14354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:44.824226 14354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:44.827155 14354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-0/data/instance:
uuid: "8c5f35ace8a54dc38cd315d50aaeb379"
format_stamp: "Formatted at 2026-06-20 18:42:44 on dist-test-slave-k6lg"
server_key: "24f92d8668e4083dc7a3fc4a341ff3a2"
server_key_iv: "b76acba1959b51619100d876cde74bc1"
server_key_version: "encryptionkey@0"
I20260620 18:42:44.827936 14354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance:
uuid: "8c5f35ace8a54dc38cd315d50aaeb379"
format_stamp: "Formatted at 2026-06-20 18:42:44 on dist-test-slave-k6lg"
server_key: "24f92d8668e4083dc7a3fc4a341ff3a2"
server_key_iv: "b76acba1959b51619100d876cde74bc1"
server_key_version: "encryptionkey@0"
I20260620 18:42:44.832931 14354 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:42:44.836449 14369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:44.837764 14354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:42:44.837939 14354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-0/wal
uuid: "8c5f35ace8a54dc38cd315d50aaeb379"
format_stamp: "Formatted at 2026-06-20 18:42:44 on dist-test-slave-k6lg"
server_key: "24f92d8668e4083dc7a3fc4a341ff3a2"
server_key_iv: "b76acba1959b51619100d876cde74bc1"
server_key_version: "encryptionkey@0"
I20260620 18:42:44.838141 14354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:44.858331 14354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:44.862142 14354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:44.862507 14354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:44.863525 14354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:44.864905 14354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:44.865003 14354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:44.865149 14354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:44.865218 14354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:44.897921 14354 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.129:46665
I20260620 18:42:44.897957 14481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.129:46665 every 8 connection(s)
I20260620 18:42:44.899205 14354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
I20260620 18:42:44.901360 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 14354
I20260620 18:42:44.901576 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance
I20260620 18:42:44.902413 6022 external_mini_cluster.cc:1428] Setting key 0ed307ac42ce2217ed89d6601e35d988
Jun 20 18:42:44 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980964, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:42:44.921550 14484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:44.901715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:45559 (local address 127.5.225.190:39687)
0620 18:42:44.902264 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:44.902270 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:44.903426 (+ 1156us) server_negotiation.cc:407] Connection header received
0620 18:42:44.905452 (+ 2026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:44.905460 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:44.905547 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:44.905667 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:44.907304 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:44.908471 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:44.910026 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:44.910613 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:44.910707 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:44.914776 (+ 4069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:44.914781 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:44.914785 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:44.914813 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:44.918089 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:44.918839 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:44.918848 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:44.918855 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:44.919011 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:44.919418 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:44.919425 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:44.919431 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:44.919836 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:44.919997 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:44.920771 (+ 774us) server_negotiation.cc:299] Negotiation successful
0620 18:42:44.921048 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":280,"threads_started":1}
I20260620 18:42:44.922224 14485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:44.902412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.129:45559)
0620 18:42:44.903256 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:44.903309 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:44.905189 (+ 1880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:44.905942 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:44.905951 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:44.906551 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:44.907107 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:44.907124 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:44.908625 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:44.908628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:44.909887 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:44.909899 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:44.910876 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:44.910886 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:44.911193 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:44.911904 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:44.911942 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:44.914562 (+ 2620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:44.918273 (+ 3711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:44.918280 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:44.918293 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:44.918700 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:44.919118 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:44.919121 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:44.919124 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:44.919298 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:44.919954 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:44.919960 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:44.920367 (+ 407us) client_negotiation.cc:769] Sending connection context
0620 18:42:44.920586 (+ 219us) client_negotiation.cc:240] Negotiation successful
0620 18:42:44.920963 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":328,"threads_started":1}
I20260620 18:42:44.923794 14482 heartbeater.cc:347] Connected to a master server at 127.5.225.190:39687
WARNING: no policy specified for kudu/127.5.225.130@KRBTEST.COM; defaulting to no policy
I20260620 18:42:44.924129 14482 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:44.924860 14482 heartbeater.cc:511] Master 127.5.225.190:39687 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.130@KRBTEST.COM" created.
I20260620 18:42:44.927228 14293 ts_manager.cc:194] Registered new tserver with Master: 8c5f35ace8a54dc38cd315d50aaeb379 (127.5.225.129:46665)
I20260620 18:42:44.928615 14293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.129@KRBTEST.COM'} at 127.5.225.129:45559
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab.
Entry for principal kudu/127.5.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab.
I20260620 18:42:44.952832 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.130:0
--local_ip_for_outbound_sockets=127.5.225.130
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:39687
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:37003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:45.273409 14488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:45.273669 14488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:45.273770 14488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:45.280853 14488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:45.280956 14488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:45.281069 14488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.130
I20260620 18:42:45.290606 14488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:37003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:39687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.130
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:45.291808 14488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:45.293023 14488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:45.307264 14494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:45.307677 14496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:45.307343 14493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:45.308593 14488 server_base.cc:1034] running on GCE node
I20260620 18:42:45.309303 14488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:45.310143 14488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:45.311395 14488 hybrid_clock.cc:648] HybridClock initialized: now 1781980965311355 us; error 32 us; skew 500 ppm
I20260620 18:42:45.311654 14488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:45 dist-test-slave-k6lg krb5kdc[14248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980965, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:45.314765 14488 init.cc:377] Logged in from keytab as kudu/127.5.225.130@KRBTEST.COM (short username kudu)
I20260620 18:42:45.316159 14488 webserver.cc:460] Webserver started at http://127.5.225.130:34397/ using document root <none> and password file <none>
I20260620 18:42:45.316910 14488 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:45.317008 14488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:45.317353 14488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:45.320346 14488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-1/data/instance:
uuid: "f5cecb6b57ec4b6793132f82eeb20d42"
format_stamp: "Formatted at 2026-06-20 18:42:45 on dist-test-slave-k6lg"
server_key: "fcc5b5501a749a84f6837b747179f9cf"
server_key_iv: "44a27fa9fe133ee35eec41efa8e7829e"
server_key_version: "encryptionkey@0"
I20260620 18:42:45.321099 14488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance:
uuid: "f5cecb6b57ec4b6793132f82eeb20d42"
format_stamp: "Formatted at 2026-06-20 18:42:45 on dist-test-slave-k6lg"
server_key: "fcc5b5501a749a84f6837b747179f9cf"
server_key_iv: "44a27fa9fe133ee35eec41efa8e7829e"
server_key_version: "encryptionkey@0"
I20260620 18:42:45.326036 14488 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260620 18:42:45.330458 14503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:45.331696 14488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:42:45.331890 14488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-1/wal
uuid: "f5cecb6b57ec4b6793132f82eeb20d42"
format_stamp: "Formatted at 2026-06-20 18:42:45 on dist-test-slave-k6lg"
server_key: "fcc5b5501a749a84f6837b747179f9cf"
server_key_iv: "44a27fa9fe133ee35eec41efa8e7829e"
server_key_version: "encryptionkey@0"
I20260620 18:42:45.332072 14488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:45.360131 14488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:45.364043 14488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:45.364279 14488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:45.365242 14488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:45.366658 14488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:45.366729 14488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:45.366818 14488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:45.366861 14488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:45.400533 14488 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.130:36925
I20260620 18:42:45.400532 14615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.130:36925 every 8 connection(s)
I20260620 18:42:45.402115 14488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
I20260620 18:42:45.403628 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 14488
I20260620 18:42:45.403803 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance
I20260620 18:42:45.404217 6022 external_mini_cluster.cc:1428] Setting key d6ef9f7a305eb0aedca9515e5b53d3e5
Jun 20 18:42:45 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980965, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:42:45.423537 14484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:45.404825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:58307 (local address 127.5.225.190:39687)
0620 18:42:45.405033 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:45.405040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:45.406567 (+ 1527us) server_negotiation.cc:407] Connection header received
0620 18:42:45.408342 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:45.408350 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:45.408435 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:45.408538 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:45.410140 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:45.411305 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:45.412793 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:45.413511 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:45.413610 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:45.417520 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:45.417526 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:45.417529 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:45.417560 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:45.420259 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:45.421196 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:45.421200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:45.421202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:45.421318 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:45.421821 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:45.421824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:45.421826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:45.422080 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:45.422243 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:45.423127 (+ 884us) server_negotiation.cc:299] Negotiation successful
0620 18:42:45.423296 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 18:42:45.425155 14618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:45.405373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.130:58307)
0620 18:42:45.406392 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:45.406448 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:45.408088 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:45.408729 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:45.408736 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:45.409403 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:45.409975 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:45.409991 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:45.411465 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:45.411468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:45.412648 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:45.412658 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:45.413695 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:45.413699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:45.413904 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:45.414610 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:45.414631 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:45.417233 (+ 2602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:45.420399 (+ 3166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:45.420411 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:45.420431 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:45.421084 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:45.421426 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:45.421433 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:45.421440 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:45.421698 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:45.422226 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:45.422237 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:45.422724 (+ 487us) client_negotiation.cc:769] Sending connection context
0620 18:42:45.423061 (+ 337us) client_negotiation.cc:240] Negotiation successful
0620 18:42:45.423673 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"thread_start_us":406,"threads_started":1}
WARNING: no policy specified for kudu/127.5.225.131@KRBTEST.COM; defaulting to no policy
I20260620 18:42:45.426672 14616 heartbeater.cc:347] Connected to a master server at 127.5.225.190:39687
I20260620 18:42:45.427006 14616 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:45.427755 14616 heartbeater.cc:511] Master 127.5.225.190:39687 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.131@KRBTEST.COM" created.
I20260620 18:42:45.429406 14293 ts_manager.cc:194] Registered new tserver with Master: f5cecb6b57ec4b6793132f82eeb20d42 (127.5.225.130:36925)
I20260620 18:42:45.430735 14293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.130@KRBTEST.COM'} at 127.5.225.130:58307
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab.
Entry for principal kudu/127.5.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab.
I20260620 18:42:45.453424 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.131:0
--local_ip_for_outbound_sockets=127.5.225.131
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:39687
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:37003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:45.769661 14621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:45.769989 14621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:45.770210 14621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:45.777465 14621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:45.777611 14621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:45.777813 14621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.131
I20260620 18:42:45.787557 14621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:37003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:39687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.131
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:45.788833 14621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:45.790222 14621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:45.803107 14626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:45.803413 14627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:45.803952 14629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:45.804448 14621 server_base.cc:1034] running on GCE node
I20260620 18:42:45.805053 14621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:45.805980 14621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:45.807174 14621 hybrid_clock.cc:648] HybridClock initialized: now 1781980965807128 us; error 64 us; skew 500 ppm
I20260620 18:42:45.807497 14621 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:45 dist-test-slave-k6lg krb5kdc[14248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980965, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:45.810590 14621 init.cc:377] Logged in from keytab as kudu/127.5.225.131@KRBTEST.COM (short username kudu)
I20260620 18:42:45.812002 14621 webserver.cc:460] Webserver started at http://127.5.225.131:34313/ using document root <none> and password file <none>
I20260620 18:42:45.812618 14621 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:45.812711 14621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:45.812966 14621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:45.815798 14621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-2/data/instance:
uuid: "4bbdb4109e86421ca067d69f1ed14442"
format_stamp: "Formatted at 2026-06-20 18:42:45 on dist-test-slave-k6lg"
server_key: "93dac7f1c6a2d7abfb7b83f2fda750a6"
server_key_iv: "8b9e5503d38bbe074a6d2af2d1be6bbd"
server_key_version: "encryptionkey@0"
I20260620 18:42:45.816514 14621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance:
uuid: "4bbdb4109e86421ca067d69f1ed14442"
format_stamp: "Formatted at 2026-06-20 18:42:45 on dist-test-slave-k6lg"
server_key: "93dac7f1c6a2d7abfb7b83f2fda750a6"
server_key_iv: "8b9e5503d38bbe074a6d2af2d1be6bbd"
server_key_version: "encryptionkey@0"
I20260620 18:42:45.821602 14621 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:42:45.825014 14636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:45.826507 14621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 18:42:45.826673 14621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-2/wal
uuid: "4bbdb4109e86421ca067d69f1ed14442"
format_stamp: "Formatted at 2026-06-20 18:42:45 on dist-test-slave-k6lg"
server_key: "93dac7f1c6a2d7abfb7b83f2fda750a6"
server_key_iv: "8b9e5503d38bbe074a6d2af2d1be6bbd"
server_key_version: "encryptionkey@0"
I20260620 18:42:45.826870 14621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:45.838627 14621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:45.842533 14621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:45.842770 14621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:45.843614 14621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:45.844856 14621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:45.844924 14621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:45.845005 14621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:45.845047 14621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:45.878289 14621 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.131:39539
I20260620 18:42:45.878361 14748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.131:39539 every 8 connection(s)
I20260620 18:42:45.879573 14621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
I20260620 18:42:45.882689 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 14621
I20260620 18:42:45.882856 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance
I20260620 18:42:45.883477 6022 external_mini_cluster.cc:1428] Setting key b9f0eddbec88fd81d151a9d8d78d7a8c
Jun 20 18:42:45 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980965, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:42:45.899737 14484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:45.882060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:57055 (local address 127.5.225.190:39687)
0620 18:42:45.882268 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:45.882272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:45.883740 (+ 1468us) server_negotiation.cc:407] Connection header received
0620 18:42:45.886087 (+ 2347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:45.886091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:45.886216 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:45.886323 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:45.887788 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:45.888517 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:45.890054 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:45.890512 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:45.890606 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:45.893850 (+ 3244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:45.893855 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:45.893858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:45.893888 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:45.896288 (+ 2400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:45.896981 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:45.896989 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:45.896993 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:45.897124 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:45.897606 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:45.897610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:45.897613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:45.897995 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:45.898130 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:45.899169 (+ 1039us) server_negotiation.cc:299] Negotiation successful
0620 18:42:45.899393 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:42:45.900854 14751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:45.882750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.131:57055)
0620 18:42:45.883545 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:45.883608 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:45.885814 (+ 2206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:45.886573 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:45.886581 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:45.887082 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:45.887631 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:45.887650 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:45.888652 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:45.888656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:45.889932 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:45.889947 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:45.890689 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:45.890694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:45.890905 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:45.891613 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:45.891634 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:45.893662 (+ 2028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:45.896415 (+ 2753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:45.896421 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:45.896435 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:45.896864 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:45.897254 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:45.897259 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:45.897263 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:45.897482 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:45.898347 (+ 865us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:45.898360 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:45.898737 (+ 377us) client_negotiation.cc:769] Sending connection context
0620 18:42:45.899073 (+ 336us) client_negotiation.cc:240] Negotiation successful
0620 18:42:45.899611 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":316,"threads_started":1}
I20260620 18:42:45.902262 14749 heartbeater.cc:347] Connected to a master server at 127.5.225.190:39687
I20260620 18:42:45.902575 14749 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:45.903252 14749 heartbeater.cc:511] Master 127.5.225.190:39687 requested a full tablet report, sending...
I20260620 18:42:45.904765 14293 ts_manager.cc:194] Registered new tserver with Master: 4bbdb4109e86421ca067d69f1ed14442 (127.5.225.131:39539)
I20260620 18:42:45.905524 14293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.131@KRBTEST.COM'} at 127.5.225.131:57055
I20260620 18:42:45.911317 6022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.5.225.129@KRBTEST.COM; defaulting to no policy
I20260620 18:42:45.931509 14482 heartbeater.cc:503] Master 127.5.225.190:39687 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.129.keytab.
Entry for principal oryx/127.5.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.129.keytab.
WARNING: no policy specified for oryx/127.5.225.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.225.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.130.keytab.
Entry for principal oryx/127.5.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.130.keytab.
WARNING: no policy specified for oryx/127.5.225.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.225.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.131.keytab.
Entry for principal oryx/127.5.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.131.keytab.
I20260620 18:42:46.049669 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 14354
Jun 20 18:42:46 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.129@KRBTEST.COM
I20260620 18:42:46.066475 14759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:46.050861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45660 (local address 127.5.225.129:46665)
0620 18:42:46.051539 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:46.051547 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:46.051574 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:42:46.051635 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:46.051641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:46.051855 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:46.051994 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:46.053282 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:46.054378 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:46.055585 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:46.056126 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:46.056220 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:46.060104 (+ 3884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:46.060116 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:46.060130 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:46.060165 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:46.063616 (+ 3451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:46.064479 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:46.064487 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:46.064489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:46.064686 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:46.065031 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:46.065035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:46.065037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:46.065369 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:46.065588 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:46.065920 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 18:42:46.066108 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":348,"threads_started":1}
I20260620 18:42:46.067698 14477 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:46.151829 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 14354
I20260620 18:42:46.168779 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 14488
Jun 20 18:42:46 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.130@KRBTEST.COM
I20260620 18:42:46.184233 14761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:46.169641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53672 (local address 127.5.225.130:36925)
0620 18:42:46.170267 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:46.170274 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:46.170301 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:42:46.170361 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:46.170368 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:46.170592 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:46.170722 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:46.171887 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:46.172900 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:46.174091 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:46.174738 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:46.174840 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:46.178934 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:46.178946 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:46.178961 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:46.178995 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:46.181874 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:46.182574 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:46.182581 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:46.182583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:46.182705 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:46.182985 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:46.182988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:46.182990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:46.183297 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:46.183502 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:46.183732 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 18:42:46.183924 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":295,"threads_started":1}
I20260620 18:42:46.185528 14611 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:46.270282 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 14488
I20260620 18:42:46.286345 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 14621
Jun 20 18:42:46 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.131@KRBTEST.COM
I20260620 18:42:46.303216 14763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:46.287366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35528 (local address 127.5.225.131:39539)
0620 18:42:46.287975 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:46.287983 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:46.288011 (+ 28us) server_negotiation.cc:407] Connection header received
0620 18:42:46.288068 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:46.288074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:46.288307 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:46.288485 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:46.289787 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:46.290952 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:46.292228 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:46.292817 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:46.292963 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:46.297600 (+ 4637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:46.297612 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:46.297628 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:46.297663 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:46.300484 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:46.301172 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:46.301179 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:46.301181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:46.301311 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:46.301673 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:46.301678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:46.301681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:46.302025 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:46.302386 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:46.302620 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 18:42:46.302885 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":306,"threads_started":1}
I20260620 18:42:46.304560 14744 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:46.388576 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 14621
I20260620 18:42:46.404575 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 14261
I20260620 18:42:46.405807 14323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:46.507386 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 14261
I20260620 18:42:46.527782 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:39687
--webserver_interface=127.5.225.190
--webserver_port=39053
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:37003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:39687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:46.845382 14766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:46.845685 14766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:46.845779 14766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:46.853147 14766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:42:46.853263 14766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:46.853310 14766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:46.853364 14766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:42:46.853412 14766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:42:46.863406 14766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:37003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:39687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:39687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=39053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:46.864703 14766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:46.865938 14766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:46.877718 14774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:46.877751 14772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:46.877897 14771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:46.878571 14766 server_base.cc:1034] running on GCE node
I20260620 18:42:46.879293 14766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:46.880625 14766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:46.881840 14766 hybrid_clock.cc:648] HybridClock initialized: now 1781980966881799 us; error 29 us; skew 500 ppm
I20260620 18:42:46.882275 14766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:46 dist-test-slave-k6lg krb5kdc[14248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980966, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:46.885385 14766 init.cc:377] Logged in from keytab as kudu/127.5.225.190@KRBTEST.COM (short username kudu)
I20260620 18:42:46.887049 14766 webserver.cc:460] Webserver started at http://127.5.225.190:39053/ using document root <none> and password file <none>
I20260620 18:42:46.887735 14766 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:46.887898 14766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:46.893054 14766 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 18:42:46.895728 14781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:46.897047 14766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:42:46.897209 14766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "8a3de271691d4d81b5faa32a51a8fae0"
format_stamp: "Formatted at 2026-06-20 18:42:44 on dist-test-slave-k6lg"
server_key: "e53c3d8a92e4ccac898436bca708cd18"
server_key_iv: "d1fa8cde9f23dc1ad867516912d167a1"
server_key_version: "encryptionkey@0"
I20260620 18:42:46.898123 14766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:46.921305 14766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:46.925231 14766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:46.925504 14766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:46.943900 14766 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:39687
I20260620 18:42:46.943914 14832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:39687 every 8 connection(s)
I20260620 18:42:46.945375 14766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:42:46.947973 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 14766
I20260620 18:42:46.948755 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.129:46665
--local_ip_for_outbound_sockets=127.5.225.129
--tserver_master_addrs=127.5.225.190:39687
--webserver_port=36775
--webserver_interface=127.5.225.129
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:37003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.225.129
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:42:46.952193 14833 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:42:46.955500 14833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0: Bootstrap starting.
I20260620 18:42:46.982585 14833 log.cc:824] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:46.993350 14833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:42:46.993956 14833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0: Bootstrap complete.
I20260620 18:42:47.007462 14833 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a3de271691d4d81b5faa32a51a8fae0" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39687 } }
I20260620 18:42:47.008754 14833 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a3de271691d4d81b5faa32a51a8fae0, State: Initialized, Role: FOLLOWER
I20260620 18:42:47.009519 14833 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a3de271691d4d81b5faa32a51a8fae0" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39687 } }
I20260620 18:42:47.009832 14833 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:42:47.009953 14833 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:42:47.010121 14833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0 [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:42:47.016425 14833 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a3de271691d4d81b5faa32a51a8fae0" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39687 } }
I20260620 18:42:47.017115 14833 leader_election.cc:302] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a3de271691d4d81b5faa32a51a8fae0; no voters:
I20260620 18:42:47.017733 14833 leader_election.cc:288] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 18:42:47.017908 14837 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 18:42:47.019575 14837 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0 [term 2 LEADER]: Becoming Leader. State: Replica: 8a3de271691d4d81b5faa32a51a8fae0, State: Running, Role: LEADER
I20260620 18:42:47.020126 14833 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:42:47.020285 14837 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a3de271691d4d81b5faa32a51a8fae0" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39687 } }
I20260620 18:42:47.022783 14839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a3de271691d4d81b5faa32a51a8fae0. Latest consensus state: current_term: 2 leader_uuid: "8a3de271691d4d81b5faa32a51a8fae0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a3de271691d4d81b5faa32a51a8fae0" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39687 } } }
I20260620 18:42:47.023187 14839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0 [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:47.023823 14838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8a3de271691d4d81b5faa32a51a8fae0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a3de271691d4d81b5faa32a51a8fae0" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39687 } } }
I20260620 18:42:47.024187 14838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0 [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:47.031549 14843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:42:47.037545 14843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:42:47.039031 14843 catalog_manager.cc:1259] Loaded cluster ID: 363a991473af4483a4cd9ff2b3e479e3
I20260620 18:42:47.039181 14843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:42:47.043205 14843 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:42:47.044752 14843 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8a3de271691d4d81b5faa32a51a8fae0: Loaded TSK: 0
I20260620 18:42:47.046375 14843 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 18:42:47.305593 14835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:47.305867 14835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:47.306010 14835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:47.313824 14835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:47.313925 14835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:47.314098 14835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.129
I20260620 18:42:47.323998 14835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:37003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.129.keytab
--principal=oryx/127.5.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.129:46665
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.225.129
--webserver_port=36775
--tserver_master_addrs=127.5.225.190:39687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.129
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:47.325315 14835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:47.326701 14835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:47.343007 14861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:47.343549 14863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:47.343875 14860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:47.344071 14835 server_base.cc:1034] running on GCE node
I20260620 18:42:47.344843 14835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:47.345979 14835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:47.347216 14835 hybrid_clock.cc:648] HybridClock initialized: now 1781980967347142 us; error 63 us; skew 500 ppm
I20260620 18:42:47.347565 14835 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:47 dist-test-slave-k6lg krb5kdc[14248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980967, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:47.351402 14835 init.cc:377] Logged in from keytab as oryx/127.5.225.129@KRBTEST.COM (short username oryx)
I20260620 18:42:47.353137 14835 webserver.cc:460] Webserver started at http://127.5.225.129:36775/ using document root <none> and password file <none>
I20260620 18:42:47.353994 14835 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:47.354143 14835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:47.361133 14835 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260620 18:42:47.364599 14870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:47.366298 14835 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:42:47.366469 14835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-0/wal
uuid: "8c5f35ace8a54dc38cd315d50aaeb379"
format_stamp: "Formatted at 2026-06-20 18:42:44 on dist-test-slave-k6lg"
server_key: "24f92d8668e4083dc7a3fc4a341ff3a2"
server_key_iv: "b76acba1959b51619100d876cde74bc1"
server_key_version: "encryptionkey@0"
I20260620 18:42:47.367883 14835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:47.407406 14835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:47.411476 14835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:47.411700 14835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:47.412627 14835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:47.413913 14835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:47.414000 14835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:47.414080 14835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:47.414145 14835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:47.445960 14835 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.129:46665
I20260620 18:42:47.446005 14982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.129:46665 every 8 connection(s)
I20260620 18:42:47.447427 14835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
I20260620 18:42:47.455889 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 14835
I20260620 18:42:47.456482 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.130:36925
--local_ip_for_outbound_sockets=127.5.225.130
--tserver_master_addrs=127.5.225.190:39687
--webserver_port=34397
--webserver_interface=127.5.225.130
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:37003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.225.130
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:42:47 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.129@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:42:47 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.129@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 18:42:47.780472 14987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:47.780781 14987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:47.780944 14987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:47.788399 14987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:47.788582 14987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:47.788766 14987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.130
I20260620 18:42:47.798820 14987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:37003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.130.keytab
--principal=oryx/127.5.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.130:36925
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.225.130
--webserver_port=34397
--tserver_master_addrs=127.5.225.190:39687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.130
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:42:47.800228 14987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:47.801679 14987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:47.816952 14994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:47.816977 14993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:47.817580 14996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:47.817842 14987 server_base.cc:1034] running on GCE node
I20260620 18:42:47.818516 14987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:47.819847 14987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:47.821069 14987 hybrid_clock.cc:648] HybridClock initialized: now 1781980967821055 us; error 100 us; skew 500 ppm
I20260620 18:42:47.821388 14987 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:47 dist-test-slave-k6lg krb5kdc[14248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980967, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:47.825168 14987 init.cc:377] Logged in from keytab as oryx/127.5.225.130@KRBTEST.COM (short username oryx)
I20260620 18:42:47.826893 14987 webserver.cc:460] Webserver started at http://127.5.225.130:34397/ using document root <none> and password file <none>
I20260620 18:42:47.827801 14987 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:47.827955 14987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:47.834239 14987 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 18:42:47.837352 15003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:47.838984 14987 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:42:47.839157 14987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-1/wal
uuid: "f5cecb6b57ec4b6793132f82eeb20d42"
format_stamp: "Formatted at 2026-06-20 18:42:45 on dist-test-slave-k6lg"
server_key: "fcc5b5501a749a84f6837b747179f9cf"
server_key_iv: "44a27fa9fe133ee35eec41efa8e7829e"
server_key_version: "encryptionkey@0"
I20260620 18:42:47.840078 14987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:47.891741 14987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:47.896137 14987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:47.896413 14987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:47.897447 14987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:47.898875 14987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:47.899003 14987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:47.899134 14987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:47.899220 14987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:47.940439 14987 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.130:36925
I20260620 18:42:47.940485 15115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.130:36925 every 8 connection(s)
I20260620 18:42:47.941860 14987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
I20260620 18:42:47.942675 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 14987
I20260620 18:42:47.943660 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.131:39539
--local_ip_for_outbound_sockets=127.5.225.131
--tserver_master_addrs=127.5.225.190:39687
--webserver_port=34313
--webserver_interface=127.5.225.131
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:37003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.225.131
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:42:47 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.130@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:42:47 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.130@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 18:42:48.591531 14986 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.130s user 0.003s sys 0.015s
W20260620 18:42:48.594146 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:47.450404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:50453 (local address 127.5.225.190:39687)
0620 18:42:47.451166 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:47.451178 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:47.451527 (+ 349us) server_negotiation.cc:407] Connection header received
0620 18:42:47.453609 (+ 2082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:47.453622 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:47.453953 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:47.454340 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:47.455927 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:47.457369 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:47.459894 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:47.460550 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:47.460687 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:48.593214 (+1132527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.129:50453: BlockingRecv error: recv got EOF from 127.5.225.129:50453 (error 108)
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":326,"threads_started":1}
I20260620 18:42:48.594357 14986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:47.450558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.129:50453)
0620 18:42:47.451347 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:47.451398 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:47.453083 (+ 1685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:47.454446 (+ 1363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:47.454453 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:47.454955 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:47.455683 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:47.455711 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:47.457558 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:47.457567 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:47.459723 (+ 2156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:47.459740 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:47.460805 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:47.460815 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:47.461124 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:47.461852 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:47.461885 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:48.592792 (+1130907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":344,"threads_started":1}
W20260620 18:42:48.595286 14983 heartbeater.cc:644] Failed to heartbeat to 127.5.225.190:39687 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.225.190:39687: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
W20260620 18:42:48.598099 15120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:48.598471 15120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:48.598646 15120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
Jun 20 18:42:48 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.129@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 18:42:48.607806 15120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:48.607937 15120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:48.608096 15120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.131
Jun 20 18:42:48 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.129@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 18:42:48.610297 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:48.596276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:47145 (local address 127.5.225.190:39687)
0620 18:42:48.596480 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:48.596487 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:48.596678 (+ 191us) server_negotiation.cc:407] Connection header received
0620 18:42:48.596919 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:48.596925 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:48.596999 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:48.597107 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:48.598409 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:48.599567 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:48.602371 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:48.603028 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:48.603160 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:48.610002 (+ 6842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.129:47145: BlockingRecv error: recv got EOF from 127.5.225.129:47145 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 18:42:48.610286 14986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:48.595961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.129:47145)
0620 18:42:48.596557 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:48.596586 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:48.596772 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:48.597186 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:48.597194 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:48.597885 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:48.598224 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:48.598241 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:48.599756 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:48.599766 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:48.602098 (+ 2332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:48.602129 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:48.603290 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:48.603300 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:48.603566 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:48.604427 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:48.604463 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:48.609810 (+ 5347us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418}
I20260620 18:42:48.620909 15120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:37003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/krb5kdc/oryx_127.5.225.131.keytab
--principal=oryx/127.5.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.131:39539
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.225.131
--webserver_port=34313
--tserver_master_addrs=127.5.225.190:39687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.131
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
Jun 20 18:42:48 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.129@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:42:48.622602 15120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
Jun 20 18:42:48 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.129@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:42:48.624298 15120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 18:42:48.625048 14986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:48.611240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.129:48783)
0620 18:42:48.611446 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:48.611473 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:48.611614 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:48.612006 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:48.612014 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:48.612605 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:48.613034 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:48.613052 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:48.614547 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:48.614555 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:48.616856 (+ 2301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:48.616878 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:48.617885 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:48.617894 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:48.618241 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:48.619054 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:48.619090 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:48.624621 (+ 5531us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 18:42:48.625589 14983 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:42:48.625157 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:48.611482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:48783 (local address 127.5.225.190:39687)
0620 18:42:48.611641 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:48.611647 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:48.611690 (+ 43us) server_negotiation.cc:407] Connection header received
0620 18:42:48.611745 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:48.611751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:48.611835 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:48.611930 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:48.613250 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:48.614355 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:48.617042 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:48.617691 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:48.617811 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:48.624858 (+ 7047us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.129:48783: BlockingRecv error: recv got EOF from 127.5.225.129:48783 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 18:42:48.639195 15126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:48.639050 15127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:48.639082 15129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:48.640059 15120 server_base.cc:1034] running on GCE node
I20260620 18:42:48.641114 15120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:48.642776 15120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:48.644119 15120 hybrid_clock.cc:648] HybridClock initialized: now 1781980968644024 us; error 59 us; skew 500 ppm
I20260620 18:42:48.644773 15120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:48 dist-test-slave-k6lg krb5kdc[14248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980968, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:48.650363 15120 init.cc:377] Logged in from keytab as oryx/127.5.225.131@KRBTEST.COM (short username oryx)
I20260620 18:42:48.652876 15120 webserver.cc:460] Webserver started at http://127.5.225.131:34313/ using document root <none> and password file <none>
I20260620 18:42:48.654215 15120 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:48.654431 15120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:48.663898 15120 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260620 18:42:48.666764 15136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:48.668227 15120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:42:48.668403 15120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-2/wal
uuid: "4bbdb4109e86421ca067d69f1ed14442"
format_stamp: "Formatted at 2026-06-20 18:42:45 on dist-test-slave-k6lg"
server_key: "93dac7f1c6a2d7abfb7b83f2fda750a6"
server_key_iv: "8b9e5503d38bbe074a6d2af2d1be6bbd"
server_key_version: "encryptionkey@0"
I20260620 18:42:48.669332 15120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:48.699983 15120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:48.707283 15120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:48.707688 15120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:48.709244 15120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:48.711637 15120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:48.711786 15120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:48.711933 15120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:48.712050 15120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:48.752123 15120 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.131:39539
I20260620 18:42:48.752118 15248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.131:39539 every 8 connection(s)
I20260620 18:42:48.754833 15120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
I20260620 18:42:48.760785 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 15120
Jun 20 18:42:48 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.131@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:42:48 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.131@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:42:48.799711 15252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:48.761861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43116 (local address 127.5.225.190:39687)
0620 18:42:48.777346 (+ 15485us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:48.777350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:48.777365 (+ 15us) server_negotiation.cc:407] Connection header received
0620 18:42:48.777403 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:48.777406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:48.777458 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:48.777585 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:48.779513 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:48.781298 (+ 1785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:48.783559 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:48.784511 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:48.784686 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:48.788522 (+ 3836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:48.788540 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:48.788576 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:48.788647 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:48.794355 (+ 5708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:48.795548 (+ 1193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:48.795563 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:48.795580 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:48.795886 (+ 306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:48.796473 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:48.796481 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:48.796487 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:48.797426 (+ 939us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:48.797861 (+ 435us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:48.798495 (+ 634us) server_negotiation.cc:299] Negotiation successful
0620 18:42:48.799051 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15382,"thread_start_us":15937,"threads_started":1}
W20260620 18:42:49.210773 15119 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.230s user 0.000s sys 0.019s
W20260620 18:42:49.212496 15118 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:47.946073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:33567 (local address 127.5.225.190:39687)
0620 18:42:47.970106 (+ 24033us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:47.970116 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:47.970147 (+ 31us) server_negotiation.cc:407] Connection header received
0620 18:42:47.970265 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:47.970273 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:47.970383 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:47.970528 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:47.972934 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:47.974248 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:47.977941 (+ 3693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:47.978774 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:47.978946 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:49.212235 (+1233289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.130:33567: BlockingRecv error: recv got EOF from 127.5.225.130:33567 (error 108)
Metrics: {"server-negotiator.queue_time_us":23823,"thread_start_us":24570,"threads_started":1}
I20260620 18:42:49.213744 15119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:47.947094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.130:33567)
0620 18:42:47.954769 (+ 7675us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:47.954820 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:47.957260 (+ 2440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:47.970832 (+ 13572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:47.970845 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:47.971569 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:47.972622 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:47.972655 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:47.974476 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:47.974485 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:47.977714 (+ 3229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:47.977741 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:47.979103 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:47.979114 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:47.979552 (+ 438us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:47.980626 (+ 1074us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:47.980696 (+ 70us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:49.212081 (+1231385us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":7311,"thread_start_us":11728,"threads_started":1}
W20260620 18:42:49.214885 15116 heartbeater.cc:644] Failed to heartbeat to 127.5.225.190:39687 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.225.190:39687: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Jun 20 18:42:49 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.130@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:42:49 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.130@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 18:42:49.229699 15118 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:49.215880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:52071 (local address 127.5.225.190:39687)
0620 18:42:49.216040 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:49.216046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:49.216068 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:42:49.216266 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:49.216271 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:49.216355 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:49.216465 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:49.217220 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:49.217978 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:49.219285 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:49.219687 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:49.219824 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:49.229417 (+ 9593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.130:52071: BlockingRecv error: recv got EOF from 127.5.225.130:52071 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 18:42:49.229765 15119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:49.215639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.130:52071)
0620 18:42:49.215898 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:49.215926 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:49.216105 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:49.216496 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:49.216501 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:49.216920 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:49.217081 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:49.217089 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:49.218120 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:49.218126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:49.219137 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:49.219148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:49.219856 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:49.219861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:49.220025 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:49.224743 (+ 4718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:49.224764 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:49.229402 (+ 4638us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":61}
Jun 20 18:42:49 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.130@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:42:49 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.130@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 18:42:49.241479 15118 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:49.230828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:59611 (local address 127.5.225.190:39687)
0620 18:42:49.231005 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:49.231010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:49.231024 (+ 14us) server_negotiation.cc:407] Connection header received
0620 18:42:49.231068 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:49.231071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:49.231141 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:49.231254 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:49.232004 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:49.232892 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:49.234296 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:49.234767 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:49.234917 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:49.241158 (+ 6241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.130:59611: BlockingRecv error: recv got EOF from 127.5.225.130:59611 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 18:42:49.241894 15119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:49.230538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.130:59611)
0620 18:42:49.230716 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:49.230739 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:49.230853 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:49.231293 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:49.231299 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:49.231675 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:49.231849 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:49.231859 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:49.233048 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:49.233054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:49.234105 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:49.234129 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:49.234942 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:49.234948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:49.235165 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:49.235946 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:49.235965 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:49.241698 (+ 5733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 18:42:49.242215 15116 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 18:42:49 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.129@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:42:49 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.129@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 18:42:49.641110 15118 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:49.626257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:47339 (local address 127.5.225.190:39687)
0620 18:42:49.626506 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:49.626517 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:49.629789 (+ 3272us) server_negotiation.cc:407] Connection header received
0620 18:42:49.629945 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:49.629953 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:49.630104 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:49.630285 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:49.631127 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:49.632808 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:49.634313 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:49.634871 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:49.635026 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:49.640922 (+ 5896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.129:47339: BlockingRecv error: recv got EOF from 127.5.225.129:47339 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:42:49.641270 15255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:49.626095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.129:47339)
0620 18:42:49.629678 (+ 3583us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:49.629694 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:49.629814 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:49.630414 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:49.630417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:49.630826 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:49.630970 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:49.630978 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:49.632978 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:49.632983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:49.634051 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:49.634063 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:49.635065 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:49.635081 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:49.635299 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:49.636362 (+ 1063us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:49.636386 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:49.640761 (+ 4375us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":3465,"thread_start_us":3312,"threads_started":1}
W20260620 18:42:49.891273 15251 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.116s user 0.008s sys 0.008s
W20260620 18:42:49.892395 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:48.758334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:36497 (local address 127.5.225.190:39687)
0620 18:42:48.758506 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:48.758510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:48.760505 (+ 1995us) server_negotiation.cc:407] Connection header received
0620 18:42:48.764162 (+ 3657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:48.764172 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:48.764286 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:48.764379 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:48.767375 (+ 2996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:48.768854 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:48.772054 (+ 3200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:48.772949 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:48.773054 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:49.892138 (+1119084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.131:36497: BlockingRecv error: recv got EOF from 127.5.225.131:36497 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:42:49.892933 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:48.759379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.131:36497)
0620 18:42:48.760285 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:48.760375 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:48.763727 (+ 3352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:48.765065 (+ 1338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:48.765081 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:48.766069 (+ 988us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:48.767112 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:48.767142 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:48.769124 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:48.769132 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:48.771840 (+ 2708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:48.771859 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:48.773451 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:48.773461 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:48.773851 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:48.775330 (+ 1479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:48.775370 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:49.891992 (+1116622us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":408,"threads_started":1}
W20260620 18:42:49.893759 15249 heartbeater.cc:644] Failed to heartbeat to 127.5.225.190:39687 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.225.190:39687: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Jun 20 18:42:49 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.131@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:42:49 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.131@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:42:49.902988 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:49.894434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.131:35699)
0620 18:42:49.894638 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:49.894658 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:49.894800 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:49.895125 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:49.895129 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:49.895489 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:49.895640 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:49.895648 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:49.896681 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:49.896687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:49.897636 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:49.897647 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:49.898453 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:49.898459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:49.898627 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:49.899185 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:49.899202 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:49.902702 (+ 3500us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 18:42:49.903159 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:49.894499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:35699 (local address 127.5.225.190:39687)
0620 18:42:49.894731 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:49.894736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:49.894753 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:42:49.894936 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:49.894945 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:49.895002 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:49.895101 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:49.895780 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:49.896547 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:49.897774 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:49.898280 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:49.898417 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:49.902897 (+ 4480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.131:35699: BlockingRecv error: recv got EOF from 127.5.225.131:35699 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
Jun 20 18:42:49 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.131@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:42:49 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.131@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:42:49.911370 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:49.903715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.131:51015)
0620 18:42:49.903883 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:49.903902 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:49.903997 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:49.904293 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:49.904297 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:49.904592 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:49.904727 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:49.904734 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:49.905676 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:49.905681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:49.906691 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:49.906702 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:49.907375 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:49.907381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:49.907570 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:49.908051 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:49.908066 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:49.911188 (+ 3122us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 18:42:49.911685 15249 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:42:49.911720 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:49.903794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:51015 (local address 127.5.225.190:39687)
0620 18:42:49.903946 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:49.903951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:49.903967 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:42:49.904089 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:49.904093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:49.904158 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:49.904258 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:49.904855 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:49.905546 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:49.906829 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:49.907231 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:49.907362 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:49.911443 (+ 4081us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.131:51015: BlockingRecv error: recv got EOF from 127.5.225.131:51015 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 18:42:50 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.130@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:42:50 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.130@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:42:50.251899 15256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:50.242810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.130:44275)
0620 18:42:50.243318 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:50.243334 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:50.243493 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:50.243898 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:50.243902 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:50.244343 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:50.244487 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:50.244495 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:50.245560 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:50.245566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:50.246611 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:50.246623 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:50.247346 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:50.247353 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:50.247511 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:50.248096 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:50.248114 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:50.251575 (+ 3461us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":300,"threads_started":1}
W20260620 18:42:50.252110 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:50.242866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:44275 (local address 127.5.225.190:39687)
0620 18:42:50.243079 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:50.243088 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:50.243419 (+ 331us) server_negotiation.cc:407] Connection header received
0620 18:42:50.243657 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:50.243664 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:50.243763 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:50.243867 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:50.244631 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:50.245420 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:50.246754 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:50.247169 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:50.247311 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:50.251858 (+ 4547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.130:44275: BlockingRecv error: recv got EOF from 127.5.225.130:44275 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 18:42:50 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.129@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:42:50 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.129@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:42:50.651105 15257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:50.642346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.129:48373)
0620 18:42:50.642899 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:50.642932 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:50.643094 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:50.643441 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:50.643446 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:50.643882 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:50.644058 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:50.644065 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:50.645038 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:50.645046 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:50.645970 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:50.645981 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:50.646784 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:50.646791 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:50.646964 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:50.647501 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:50.647517 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:50.650876 (+ 3359us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":333,"threads_started":1}
W20260620 18:42:50.651312 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:50.642372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:48373 (local address 127.5.225.190:39687)
0620 18:42:50.642572 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:50.642578 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:50.643015 (+ 437us) server_negotiation.cc:407] Connection header received
0620 18:42:50.643228 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:50.643233 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:50.643312 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:50.643413 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:50.644197 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:50.644907 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:50.646105 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:50.646617 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:50.646752 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:50.651046 (+ 4294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.129:48373: BlockingRecv error: recv got EOF from 127.5.225.129:48373 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 18:42:50 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.131@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:42:50 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.131@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:42:50.921382 15258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:50.912257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.131:46105)
0620 18:42:50.912741 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:50.912757 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:50.912887 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:50.913161 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:50.913164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:50.913617 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:50.913820 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:50.913830 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:50.915186 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:50.915190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:50.916187 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:50.916199 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:50.917105 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:50.917110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:50.917219 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:50.917833 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:50.917850 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:50.921150 (+ 3300us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":280,"threads_started":1}
W20260620 18:42:50.921705 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:50.912330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:46105 (local address 127.5.225.190:39687)
0620 18:42:50.912571 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:50.912578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:50.912858 (+ 280us) server_negotiation.cc:407] Connection header received
0620 18:42:50.912984 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:50.912988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:50.913048 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:50.913125 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:50.913973 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:50.915069 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:50.916333 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:50.916953 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:50.917045 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:50.921373 (+ 4328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.131:46105: BlockingRecv error: recv got EOF from 127.5.225.131:46105 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 20 18:42:51 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.130@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:42:51 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.130@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:42:51.263336 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:51.252918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.130:36507)
0620 18:42:51.253451 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:51.253468 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:51.253598 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:51.253880 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:51.253883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:51.254420 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:51.254746 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:51.254763 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:51.256184 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:51.256189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:51.257116 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:51.257127 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:51.257804 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:51.257810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:51.257992 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:51.258662 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:51.258687 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:51.262967 (+ 4280us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":300,"threads_started":1}
W20260620 18:42:51.263553 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:51.252990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:36507 (local address 127.5.225.190:39687)
0620 18:42:51.253145 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:51.253150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:51.253539 (+ 389us) server_negotiation.cc:407] Connection header received
0620 18:42:51.253687 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:51.253691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:51.253749 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:51.253831 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:51.254957 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:51.256038 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:51.257257 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:51.257656 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:51.257794 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:51.263268 (+ 5474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.130:36507: BlockingRecv error: recv got EOF from 127.5.225.130:36507 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 18:42:51 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.129@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:42:51 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.129@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:42:51.661525 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:51.652008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.129:49969)
0620 18:42:51.652556 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:51.652572 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:51.652690 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:51.653012 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:51.653016 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:51.653367 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:51.653549 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:51.653558 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:51.654658 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:51.654662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:51.655622 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:51.655634 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:51.656496 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:51.656500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:51.656870 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:51.657711 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:51.657738 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:51.661275 (+ 3537us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":318,"threads_started":1}
W20260620 18:42:51.661698 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:51.652168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:49969 (local address 127.5.225.190:39687)
0620 18:42:51.652358 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:51.652363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:51.652645 (+ 282us) server_negotiation.cc:407] Connection header received
0620 18:42:51.652784 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:51.652787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:51.652845 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:51.652923 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:51.653674 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:51.654418 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:51.655751 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:51.656353 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:51.656485 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:51.661465 (+ 4980us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.129:49969: BlockingRecv error: recv got EOF from 127.5.225.129:49969 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 18:42:51 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.131@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:42:51 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.131@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:42:51.932008 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:51.922361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.131:39125)
0620 18:42:51.922853 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:51.922868 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:51.923036 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:51.923411 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:51.923415 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:51.923841 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:51.924016 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:51.924024 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:51.925085 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:51.925092 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:51.926007 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:51.926017 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:51.926774 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:51.926780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:51.926957 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:51.927612 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:51.927633 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:51.931749 (+ 4116us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":289,"threads_started":1}
W20260620 18:42:51.932261 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:51.922420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:39125 (local address 127.5.225.190:39687)
0620 18:42:51.922641 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:51.922649 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:51.922959 (+ 310us) server_negotiation.cc:407] Connection header received
0620 18:42:51.923200 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:51.923212 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:51.923274 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:51.923381 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:51.924165 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:51.924938 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:51.926155 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:51.926602 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:51.926754 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:51.931989 (+ 5235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.131:39125: BlockingRecv error: recv got EOF from 127.5.225.131:39125 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 20 18:42:52 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.130@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:42:52 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.130@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:42:52.274466 15262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:52.264333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.130:47953)
0620 18:42:52.264979 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:52.264995 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:52.265168 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:52.265544 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:52.265549 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:52.265926 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:52.266083 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:52.266092 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:52.267245 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:52.267252 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:52.268203 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:52.268215 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:52.269124 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:52.269130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:52.269308 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:52.270055 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:52.270072 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:52.274154 (+ 4082us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":378,"threads_started":1}
W20260620 18:42:52.274717 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:52.264504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:47953 (local address 127.5.225.190:39687)
0620 18:42:52.264771 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:52.264779 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:52.265092 (+ 313us) server_negotiation.cc:407] Connection header received
0620 18:42:52.265307 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:52.265312 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:52.265383 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:52.265495 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:52.266292 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:52.267043 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:52.268350 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:52.268958 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:52.269111 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:52.274397 (+ 5286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.130:47953: BlockingRecv error: recv got EOF from 127.5.225.130:47953 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Jun 20 18:42:52 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.129@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:42:52 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.129@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:42:52.671491 15263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:52.662519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.129:54533)
0620 18:42:52.663042 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:52.663058 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:52.663206 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:52.663566 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:52.663571 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:52.663966 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:52.664123 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:52.664131 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:52.665137 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:52.665145 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:52.666153 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:52.666188 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:52.666867 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:52.666873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:52.667080 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:52.667749 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:52.667771 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:52.671144 (+ 3373us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":309,"threads_started":1}
W20260620 18:42:52.671566 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:52.662584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:54533 (local address 127.5.225.190:39687)
0620 18:42:52.662804 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:52.662811 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:52.663150 (+ 339us) server_negotiation.cc:407] Connection header received
0620 18:42:52.663361 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:52.663364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:52.663424 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:52.663523 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:52.664271 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:52.665013 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:52.666325 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:52.666720 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:52.666855 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:52.671342 (+ 4487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.129:54533: BlockingRecv error: recv got EOF from 127.5.225.129:54533 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 18:42:52 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.131@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:42:52 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.131@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:42:52.941951 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:52.932970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.131:54709)
0620 18:42:52.933475 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:52.933490 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:52.933653 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:52.933983 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:52.933987 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:52.934398 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:52.934564 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:52.934573 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:52.935592 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:52.935597 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:52.936510 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:52.936521 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:52.937181 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:52.937187 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:52.937372 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:52.937994 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:52.938011 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:52.941690 (+ 3679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":265,"threads_started":1}
W20260620 18:42:52.942195 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:52.933044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:54709 (local address 127.5.225.190:39687)
0620 18:42:52.933307 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:52.933313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:52.933575 (+ 262us) server_negotiation.cc:407] Connection header received
0620 18:42:52.933784 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:52.933790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:52.933856 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:52.933951 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:52.934713 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:52.935446 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:52.936648 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:52.937035 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:52.937157 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:52.941908 (+ 4751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.131:54709: BlockingRecv error: recv got EOF from 127.5.225.131:54709 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Jun 20 18:42:53 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.130@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:42:53 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.130@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:42:53.285332 15265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:53.275736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.130:58399)
0620 18:42:53.276254 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:53.276271 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:53.276429 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:53.276817 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:53.276824 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:53.277224 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:53.277380 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:53.277389 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:53.278553 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:53.278558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:53.279822 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:53.279834 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:53.280569 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:53.280573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:53.280785 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:53.281475 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:53.281493 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:53.284936 (+ 3443us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":287,"threads_started":1}
W20260620 18:42:53.285430 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:53.275828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:58399 (local address 127.5.225.190:39687)
0620 18:42:53.276075 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:53.276083 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:53.276361 (+ 278us) server_negotiation.cc:407] Connection header received
0620 18:42:53.276563 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:53.276574 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:53.276669 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:53.276778 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:53.277523 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:53.278362 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:53.279978 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:53.280434 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:53.280526 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:53.285143 (+ 4617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.130:58399: BlockingRecv error: recv got EOF from 127.5.225.130:58399 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 20 18:42:53 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.129@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:42:53 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.129@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:42:53.682410 15266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:53.672553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.129:59713)
0620 18:42:53.673108 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:53.673147 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:53.673262 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:53.673629 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:53.673634 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:53.674042 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:53.674275 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:53.674284 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:53.675358 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:53.675362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:53.676335 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:53.676346 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:53.677298 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:53.677302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:53.677529 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:53.678258 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:53.678278 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:53.682036 (+ 3758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":317,"threads_started":1}
W20260620 18:42:53.682538 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:53.672669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:59713 (local address 127.5.225.190:39687)
0620 18:42:53.672847 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:53.672873 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:53.673229 (+ 356us) server_negotiation.cc:407] Connection header received
0620 18:42:53.673372 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:53.673376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:53.673441 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:53.673529 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:53.674403 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:53.675230 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:53.676493 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:53.677123 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:53.677326 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:53.682279 (+ 4953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.129:59713: BlockingRecv error: recv got EOF from 127.5.225.129:59713 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 18:42:53 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.131@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:42:53 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.131@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 18:42:53.954813 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:53.943366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:53883 (local address 127.5.225.190:39687)
0620 18:42:53.943577 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:53.943586 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:53.943879 (+ 293us) server_negotiation.cc:407] Connection header received
0620 18:42:53.944046 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:53.944052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:53.944142 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:53.944250 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:53.945256 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:53.946480 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:53.948472 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:53.949091 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:53.949178 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:53.954529 (+ 5351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.131:53883: BlockingRecv error: recv got EOF from 127.5.225.131:53883 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 18:42:53.955454 15267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:53.942977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.131:53883)
0620 18:42:53.943715 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:53.943788 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:53.943924 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:53.944322 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:53.944328 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:53.944871 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:53.945080 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:53.945094 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:53.946645 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:53.946653 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:53.948276 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:53.948321 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:53.949441 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:53.949447 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:53.949646 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:53.950356 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:53.950377 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:53.955128 (+ 4751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":603,"threads_started":1}
Jun 20 18:42:54 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.130@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:42:54 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.130@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:42:54.299190 15268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:54.286353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.130:52623)
0620 18:42:54.286868 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:54.286884 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:54.286996 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:54.287317 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:54.287320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:54.287717 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:54.287864 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:54.287873 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:54.288933 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:54.288936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:54.289886 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:54.289897 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:54.290874 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:54.290880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:54.291057 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:54.295543 (+ 4486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:54.295563 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:54.298974 (+ 3411us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":269,"threads_started":1}
W20260620 18:42:54.299453 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:54.286538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:52623 (local address 127.5.225.190:39687)
0620 18:42:54.286757 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:54.286761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:54.286961 (+ 200us) server_negotiation.cc:407] Connection header received
0620 18:42:54.287099 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:54.287102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:54.287182 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:54.287266 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:54.288029 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:54.288817 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:54.290044 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:54.290720 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:54.290863 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:54.299165 (+ 8302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.130:52623: BlockingRecv error: recv got EOF from 127.5.225.130:52623 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Jun 20 18:42:54 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.129@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:42:54 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.129@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:42:54.692509 15269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:54.683421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.129:60011)
0620 18:42:54.683901 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:54.683917 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:54.684071 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:54.684443 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:54.684448 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:54.684887 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:54.685063 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:54.685072 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:54.686296 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:54.686304 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:54.687287 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:54.687298 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:54.687994 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:54.688000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:54.688182 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:54.688855 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:54.688878 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:54.692185 (+ 3307us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":252,"threads_started":1}
W20260620 18:42:54.692603 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:54.683512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:60011 (local address 127.5.225.190:39687)
0620 18:42:54.683747 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:54.683754 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:54.684009 (+ 255us) server_negotiation.cc:407] Connection header received
0620 18:42:54.684204 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:54.684211 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:54.684302 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:54.684409 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:54.685210 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:54.686130 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:54.687424 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:54.687845 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:54.687980 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:54.692376 (+ 4396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.129:60011: BlockingRecv error: recv got EOF from 127.5.225.129:60011 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Jun 20 18:42:54 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.131@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:42:54 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.131@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:42:54.965076 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:54.956364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.131:37323)
0620 18:42:54.957031 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:54.957049 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:54.957170 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:54.957491 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:54.957494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:54.957886 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:54.958038 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:54.958046 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:54.959084 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:54.959088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:54.960092 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:54.960103 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:54.960788 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:54.960793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:54.960933 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:54.961510 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:54.961528 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:54.964860 (+ 3332us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":426,"threads_started":1}
W20260620 18:42:54.965274 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:54.956510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:37323 (local address 127.5.225.190:39687)
0620 18:42:54.956697 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:54.956704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:54.957142 (+ 438us) server_negotiation.cc:407] Connection header received
0620 18:42:54.957287 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:54.957291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:54.957352 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:54.957435 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:54.958209 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:54.958955 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:54.960242 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:54.960647 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:54.960767 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:54.965040 (+ 4273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.131:37323: BlockingRecv error: recv got EOF from 127.5.225.131:37323 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 18:42:55 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.130@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:42:55 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.130@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:42:55.310000 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:55.300198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.130:36035)
0620 18:42:55.301025 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:55.301048 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:55.301217 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:55.301769 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:55.301776 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:55.302220 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:55.302392 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:55.302402 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:55.303704 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:55.303708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:55.304636 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:55.304649 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:55.305381 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:55.305385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:55.305509 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:55.306133 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:55.306153 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:55.309751 (+ 3598us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":708,"thread_start_us":389,"threads_started":1}
W20260620 18:42:55.310333 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:55.300339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:36035 (local address 127.5.225.190:39687)
0620 18:42:55.300604 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:55.300611 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:55.301163 (+ 552us) server_negotiation.cc:407] Connection header received
0620 18:42:55.301365 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:55.301371 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:55.301475 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:55.301729 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:55.302566 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:55.303520 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:55.304792 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:55.305230 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:55.305336 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:55.309949 (+ 4613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.130:36035: BlockingRecv error: recv got EOF from 127.5.225.130:36035 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 20 18:42:55 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.129@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:42:55 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.129@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:42:55.702761 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:55.693443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.129:60723)
0620 18:42:55.694257 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:55.694273 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:55.694396 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:55.694661 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:55.694665 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:55.695052 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:55.695210 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:55.695218 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:55.696301 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:55.696305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:55.697212 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:55.697223 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:55.697882 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:55.697886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:55.697995 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:55.698639 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:55.698657 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:55.702404 (+ 3747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":497,"threads_started":1}
W20260620 18:42:55.702857 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:55.693573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:60723 (local address 127.5.225.190:39687)
0620 18:42:55.693803 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:55.693811 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:55.694359 (+ 548us) server_negotiation.cc:407] Connection header received
0620 18:42:55.694477 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:55.694481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:55.694541 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:55.694628 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:55.695369 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:55.696177 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:55.697346 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:55.697759 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:55.697850 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:55.702587 (+ 4737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.129:60723: BlockingRecv error: recv got EOF from 127.5.225.129:60723 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 20 18:42:55 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.131@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:42:55 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.131@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:42:55.975138 15273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:55.966088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.131:50701)
0620 18:42:55.966775 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:55.966820 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:55.966928 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:55.967331 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:55.967336 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:55.967766 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:55.967942 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:55.967951 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:55.968893 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:55.968896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:55.969800 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:55.969811 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:55.970543 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:55.970547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:55.970672 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:55.971248 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:55.971264 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:55.974817 (+ 3553us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":458,"threads_started":1}
W20260620 18:42:55.975255 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:55.966264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:50701 (local address 127.5.225.190:39687)
0620 18:42:55.966445 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:55.966451 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:55.966908 (+ 457us) server_negotiation.cc:407] Connection header received
0620 18:42:55.967075 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:55.967079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:55.967165 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:55.967267 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:55.968080 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:55.968779 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:55.969923 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:55.970413 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:55.970501 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:55.975007 (+ 4506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.131:50701: BlockingRecv error: recv got EOF from 127.5.225.131:50701 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 18:42:56 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.130@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:42:56 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.130@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:42:56.320813 15274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:56.311084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.130:59483)
0620 18:42:56.311726 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:56.311748 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:56.311863 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:56.312144 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:56.312147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:56.312509 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:56.312680 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:56.312687 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:56.313696 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:56.313701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:56.315300 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:56.315314 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:56.316285 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:56.316294 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:56.316512 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:56.317096 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:56.317113 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:56.320549 (+ 3436us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":301,"threads_started":1}
W20260620 18:42:56.321053 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:56.311139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:59483 (local address 127.5.225.190:39687)
0620 18:42:56.311329 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:56.311334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:56.311819 (+ 485us) server_negotiation.cc:407] Connection header received
0620 18:42:56.311963 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:56.311966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:56.312024 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:56.312104 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:56.312832 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:56.313554 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:56.315431 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:56.316084 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:56.316232 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:56.320785 (+ 4553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.130:59483: BlockingRecv error: recv got EOF from 127.5.225.130:59483 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 18:42:56 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.129@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:42:56 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.129@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:42:56.712478 15275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:56.703734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.129:57381)
0620 18:42:56.704277 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:56.704293 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:56.704403 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:56.704732 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:56.704735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:56.705124 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:56.705279 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:56.705289 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:56.706298 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:56.706302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:56.707232 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:56.707242 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:56.707917 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:56.707922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:56.708034 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:56.708674 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:56.708691 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:56.712244 (+ 3553us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":289,"threads_started":1}
W20260620 18:42:56.712754 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:56.703855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:57381 (local address 127.5.225.190:39687)
0620 18:42:56.704024 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:56.704028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:56.704392 (+ 364us) server_negotiation.cc:407] Connection header received
0620 18:42:56.704501 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:56.704505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:56.704576 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:56.704713 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:56.705400 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:56.706108 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:56.707357 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:56.707758 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:56.707856 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:56.712471 (+ 4615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.129:57381: BlockingRecv error: recv got EOF from 127.5.225.129:57381 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 18:42:56 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.131@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:42:56 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.131@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:42:56.984831 15276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:56.976156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.131:41117)
0620 18:42:56.976765 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:56.976787 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:56.976900 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:56.977183 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:56.977186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:56.977658 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:56.977846 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:56.977855 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:56.978897 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:56.978901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:56.979808 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:56.979819 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:56.980510 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:56.980513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:56.980621 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:56.981276 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:56.981293 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:56.984594 (+ 3301us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":324,"threads_started":1}
W20260620 18:42:56.985056 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:56.976332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:41117 (local address 127.5.225.190:39687)
0620 18:42:56.976496 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:56.976500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:56.976900 (+ 400us) server_negotiation.cc:407] Connection header received
0620 18:42:56.976998 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:56.977001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:56.977064 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:56.977161 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:56.977981 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:56.978768 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:56.979940 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:56.980383 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:56.980477 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:56.984797 (+ 4320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.131:41117: BlockingRecv error: recv got EOF from 127.5.225.131:41117 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 20 18:42:57 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.130@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:42:57 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.130@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:42:57.331184 15277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:57.321907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.130:48903)
0620 18:42:57.322530 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:57.322546 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:57.322724 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:57.323108 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:57.323112 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:57.323530 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:57.323687 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:57.323696 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:57.324876 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:57.324881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:57.325829 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:57.325841 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:57.326685 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:57.326691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:57.326870 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:57.327517 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:57.327534 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:57.330851 (+ 3317us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":345,"threads_started":1}
W20260620 18:42:57.331301 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:57.321991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:48903 (local address 127.5.225.190:39687)
0620 18:42:57.322360 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:57.322366 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:57.322639 (+ 273us) server_negotiation.cc:407] Connection header received
0620 18:42:57.322893 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:57.322897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:57.322959 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:57.323071 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:57.323846 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:57.324727 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:57.325980 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:57.326494 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:57.326639 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:57.331040 (+ 4401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.130:48903: BlockingRecv error: recv got EOF from 127.5.225.130:48903 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Jun 20 18:42:57 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.129@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:42:57 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.129@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:42:57.723031 15278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:57.713447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.129:37605)
0620 18:42:57.714019 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:57.714037 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:57.714246 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:57.714648 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:57.714654 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:57.715145 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:57.715303 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:57.715312 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:57.716416 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:57.716424 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:57.717481 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:57.717492 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:57.718334 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:57.718341 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:57.718536 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:57.719137 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:57.719156 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:57.722816 (+ 3660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":289,"threads_started":1}
W20260620 18:42:57.723258 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:57.713507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:37605 (local address 127.5.225.190:39687)
0620 18:42:57.713797 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:57.713803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:57.714147 (+ 344us) server_negotiation.cc:407] Connection header received
0620 18:42:57.714423 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:57.714427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:57.714492 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:57.714625 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:57.715458 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:57.716270 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:57.717649 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:57.718106 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:57.718304 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:57.722984 (+ 4680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.129:37605: BlockingRecv error: recv got EOF from 127.5.225.129:37605 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Jun 20 18:42:57 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.131@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:42:57 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.131@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:42:57.994828 15279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:57.985712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.131:34813)
0620 18:42:57.986322 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:57.986338 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:57.986487 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:57.986842 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:57.986846 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:57.987265 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:57.987439 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:57.987447 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:57.988455 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:57.988462 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:57.989398 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:57.989409 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:57.990374 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:57.990382 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:57.990552 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:57.991180 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:57.991197 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:57.994586 (+ 3389us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":293,"threads_started":1}
W20260620 18:42:57.994999 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:57.985806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:34813 (local address 127.5.225.190:39687)
0620 18:42:57.986037 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:57.986045 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:57.986423 (+ 378us) server_negotiation.cc:407] Connection header received
0620 18:42:57.986629 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:57.986634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:57.986707 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:57.986805 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:57.987585 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:57.988309 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:57.989541 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:57.990111 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:57.990302 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:57.994771 (+ 4469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.131:34813: BlockingRecv error: recv got EOF from 127.5.225.131:34813 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 20 18:42:58 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.130@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:42:58 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.130@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:42:58.342355 15280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:58.332442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.130:45677)
0620 18:42:58.332977 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:58.332993 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:58.333138 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:58.333483 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:58.333486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:58.333923 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:58.334054 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:58.334061 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:58.335031 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:58.335035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:58.335958 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:58.335969 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:58.336650 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:58.336655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:58.336769 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:58.337486 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:58.337503 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:58.341801 (+ 4298us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":283,"threads_started":1}
W20260620 18:42:58.342787 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:58.332495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:45677 (local address 127.5.225.190:39687)
0620 18:42:58.332651 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:58.332656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:58.333092 (+ 436us) server_negotiation.cc:407] Connection header received
0620 18:42:58.333257 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:58.333261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:58.333338 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:58.333447 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:58.334186 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:58.334916 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:58.336082 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:58.336520 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:58.336611 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:58.342544 (+ 5933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.130:45677: BlockingRecv error: recv got EOF from 127.5.225.130:45677 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 18:42:58 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.129@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:42:58 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.129@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:42:58.733156 15281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:58.724055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.129:53351)
0620 18:42:58.724540 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:58.724557 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:58.724669 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:58.725066 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:58.725070 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:58.725445 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:58.725614 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:58.725623 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:58.726662 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:58.726667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:58.727811 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:58.727824 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:58.728670 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:58.728677 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:58.728858 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:58.729561 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:58.729589 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:58.732843 (+ 3254us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":286,"threads_started":1}
W20260620 18:42:58.733330 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:58.724102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:53351 (local address 127.5.225.190:39687)
0620 18:42:58.724334 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:58.724344 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:58.724645 (+ 301us) server_negotiation.cc:407] Connection header received
0620 18:42:58.724817 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:58.724821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:58.724885 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:58.724972 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:58.725790 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:58.726526 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:58.727940 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:58.728512 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:58.728647 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:58.733038 (+ 4391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.129:53351: BlockingRecv error: recv got EOF from 127.5.225.129:53351 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 18:42:59 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.131@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:42:59 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.131@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:42:59.004655 15282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:58.995729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.131:49877)
0620 18:42:58.996256 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:58.996275 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:58.996428 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:58.996721 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:58.996725 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:58.997166 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:58.997318 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:58.997327 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:58.998381 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:58.998385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:58.999370 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:58.999383 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:59.000078 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:59.000084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:59.000217 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:59.000808 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:59.000827 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:59.004431 (+ 3604us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":261,"threads_started":1}
W20260620 18:42:59.004906 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:58.995799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:49877 (local address 127.5.225.190:39687)
0620 18:42:58.996040 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:58.996047 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:58.996376 (+ 329us) server_negotiation.cc:407] Connection header received
0620 18:42:58.996531 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:58.996534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:58.996592 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:58.996669 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:58.997440 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:58.998253 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:58.999512 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:58.999909 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:59.000002 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:59.004615 (+ 4613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.131:49877: BlockingRecv error: recv got EOF from 127.5.225.131:49877 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 18:42:59 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.130@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:42:59 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.130@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:42:59.355391 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:59.343491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.130:59721)
0620 18:42:59.344069 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:59.344085 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:59.344198 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:59.344487 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:59.344491 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:59.344936 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:59.345096 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:59.345105 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:59.346253 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:59.346257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:59.347261 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:59.347272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:59.347966 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:59.347971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:59.348075 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:59.351542 (+ 3467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:59.351559 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:59.355138 (+ 3579us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":364,"threads_started":1}
W20260620 18:42:59.355558 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:59.343545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:59721 (local address 127.5.225.190:39687)
0620 18:42:59.343888 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:59.343902 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:59.344192 (+ 290us) server_negotiation.cc:407] Connection header received
0620 18:42:59.344289 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:59.344293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:59.344362 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:59.344452 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:59.345246 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:59.346071 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:59.347388 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:59.347837 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:59.347929 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:59.355323 (+ 7394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.130:59721: BlockingRecv error: recv got EOF from 127.5.225.130:59721 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Jun 20 18:42:59 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.129@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:42:59 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.129@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:42:59.742993 15284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:59.734033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.129:57511)
0620 18:42:59.734604 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:59.734626 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:59.734793 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:59.735163 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:59.735169 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:59.735602 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:59.735779 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:59.735788 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:59.736814 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:59.736820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:59.737762 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:59.737774 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:59.738544 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:59.738551 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:59.738738 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:59.739374 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:59.739392 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:59.742649 (+ 3257us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":340,"threads_started":1}
W20260620 18:42:59.743065 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:42:59.734259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:57511 (local address 127.5.225.190:39687)
0620 18:42:59.734538 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:59.734554 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:59.734695 (+ 141us) server_negotiation.cc:407] Connection header received
0620 18:42:59.734916 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:59.734921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:59.734997 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:59.735111 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:59.735921 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:59.736681 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:59.737905 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:59.738389 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:59.738528 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:59.742827 (+ 4299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.129:57511: BlockingRecv error: recv got EOF from 127.5.225.129:57511 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
Jun 20 18:43:00 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.131@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:43:00 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.131@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:43:00.015420 15285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:00.005634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.131:42111)
0620 18:43:00.006261 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:00.006276 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:00.006436 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:00.006780 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:00.006785 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:00.007203 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:00.007362 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:00.007370 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:00.008389 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:00.008397 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:00.009632 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:00.009645 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:00.010487 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:00.010492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:00.010664 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:00.011274 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:00.011291 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:00.015148 (+ 3857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":322,"threads_started":1}
W20260620 18:43:00.015507 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:00.005840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:42111 (local address 127.5.225.190:39687)
0620 18:43:00.006092 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:00.006099 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:00.006368 (+ 269us) server_negotiation.cc:407] Connection header received
0620 18:43:00.006576 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:00.006580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:00.006640 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:00.006755 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:00.007516 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:00.008258 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:00.009840 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:00.010334 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:00.010471 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:00.015324 (+ 4853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.131:42111: BlockingRecv error: recv got EOF from 127.5.225.131:42111 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Jun 20 18:43:00 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.130@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:43:00 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.130@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:43:00.365469 15286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:00.356429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.130:44461)
0620 18:43:00.356896 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:00.356911 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:00.357066 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:00.357424 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:00.357428 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:00.357844 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:00.358016 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:00.358025 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:00.359058 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:00.359065 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:00.359991 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:00.360003 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:00.360743 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:00.360748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:00.360926 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:00.361654 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:00.361678 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:00.365246 (+ 3568us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":253,"threads_started":1}
W20260620 18:43:00.365655 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:00.356513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:44461 (local address 127.5.225.190:39687)
0620 18:43:00.356748 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:00.356755 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:00.356997 (+ 242us) server_negotiation.cc:407] Connection header received
0620 18:43:00.357196 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:00.357201 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:00.357287 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:00.357388 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:00.358180 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:00.358928 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:00.360131 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:00.360585 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:00.360714 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:00.365408 (+ 4694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.130:44461: BlockingRecv error: recv got EOF from 127.5.225.130:44461 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Jun 20 18:43:00 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.129@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:43:00 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.129@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:43:00.752947 15287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:00.744017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.129:50911)
0620 18:43:00.744517 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:00.744533 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:00.744646 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:00.745031 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:00.745036 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:00.745508 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:00.745720 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:00.745729 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:00.746903 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:00.746910 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:00.747832 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:00.747843 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:00.748560 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:00.748566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:00.748735 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:00.749333 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:00.749350 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:00.752653 (+ 3303us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":258,"threads_started":1}
W20260620 18:43:00.753139 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:00.744087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:50911 (local address 127.5.225.190:39687)
0620 18:43:00.744342 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:00.744349 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:00.744614 (+ 265us) server_negotiation.cc:407] Connection header received
0620 18:43:00.744819 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:00.744823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:00.744885 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:00.744996 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:00.745866 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:00.746760 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:00.747974 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:00.748378 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:00.748502 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:00.752901 (+ 4399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.129:50911: BlockingRecv error: recv got EOF from 127.5.225.129:50911 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 20 18:43:01 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.131@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:43:01 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.131@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:43:01.026094 15288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:01.016452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.131:50959)
0620 18:43:01.016970 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:01.016985 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:01.017144 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:01.017489 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:01.017493 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:01.017918 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:01.018131 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:01.018146 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.019626 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.019630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:01.020760 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:01.020771 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.021599 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.021604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:01.021763 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:01.022432 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:01.022449 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:01.025865 (+ 3416us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":297,"threads_started":1}
W20260620 18:43:01.026328 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:01.016551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:50959 (local address 127.5.225.190:39687)
0620 18:43:01.016827 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:01.016835 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:01.017070 (+ 235us) server_negotiation.cc:407] Connection header received
0620 18:43:01.017280 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:01.017287 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:01.017355 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:01.017458 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:01.018353 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.019477 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.020913 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.021464 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.021568 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:01.026041 (+ 4473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.131:50959: BlockingRecv error: recv got EOF from 127.5.225.131:50959 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Jun 20 18:43:01 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.130@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:43:01 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.130@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:43:01.375909 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:01.366469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.130:50485)
0620 18:43:01.367032 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:01.367051 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:01.367199 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:01.367592 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:01.367595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:01.368092 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:01.368360 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:01.368373 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.369490 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.369494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:01.370519 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:01.370530 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.371277 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.371281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:01.371402 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:01.372048 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:01.372064 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:01.375676 (+ 3612us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":304,"threads_started":1}
W20260620 18:43:01.376116 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:01.366556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:50485 (local address 127.5.225.190:39687)
0620 18:43:01.366794 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:01.366803 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:01.367169 (+ 366us) server_negotiation.cc:407] Connection header received
0620 18:43:01.367355 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:01.367367 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:01.367446 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:01.367544 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:01.368512 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.369356 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.370642 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.371115 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.371207 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:01.375865 (+ 4658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.130:50485: BlockingRecv error: recv got EOF from 127.5.225.130:50485 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 20 18:43:01 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.129@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:43:01 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.129@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:43:01.763641 15290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:01.754134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.129:54145)
0620 18:43:01.754703 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:01.754720 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:01.754892 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:01.755242 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:01.755247 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:01.755684 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:01.755831 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:01.755839 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.756875 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.756882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:01.757812 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:01.757822 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.758642 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.758647 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:01.758834 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:01.759495 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:01.759513 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:01.763317 (+ 3804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":296,"threads_started":1}
W20260620 18:43:01.763749 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:01.754300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:54145 (local address 127.5.225.190:39687)
0620 18:43:01.754504 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:01.754513 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:01.754822 (+ 309us) server_negotiation.cc:407] Connection header received
0620 18:43:01.755029 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:01.755034 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:01.755102 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:01.755211 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:01.755985 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.756729 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.757965 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.758478 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.758635 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:01.763512 (+ 4877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.129:54145: BlockingRecv error: recv got EOF from 127.5.225.129:54145 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 18:43:02 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.131@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:43:02 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.131@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:43:02.036950 15291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:02.027140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.131:50499)
0620 18:43:02.027770 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:02.027787 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:02.027902 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:02.028206 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:02.028209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:02.028582 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:02.028757 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:02.028765 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.029885 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.029890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:02.030923 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:02.030935 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.031643 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.031647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:02.031838 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:02.032573 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:02.032602 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:02.036666 (+ 4064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":327,"threads_started":1}
W20260620 18:43:02.037055 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:02.027166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:50499 (local address 127.5.225.190:39687)
0620 18:43:02.027315 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:02.027319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:02.027898 (+ 579us) server_negotiation.cc:407] Connection header received
0620 18:43:02.027990 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:02.027994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:02.028063 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:02.028156 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:02.028914 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.029739 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.031060 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.031482 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.031583 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:02.036881 (+ 5298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.131:50499: BlockingRecv error: recv got EOF from 127.5.225.131:50499 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 18:43:02 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.130@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:43:02 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.130@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:43:02.386266 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:02.376880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.130:50267)
0620 18:43:02.377466 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:02.377481 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:02.377645 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:02.378059 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:02.378065 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:02.378549 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:02.378701 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:02.378711 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.379724 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.379730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:02.380618 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:02.380630 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.381423 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.381428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:02.381607 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:02.382271 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:02.382289 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:02.385949 (+ 3660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":364,"threads_started":1}
W20260620 18:43:02.386482 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:02.377081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:50267 (local address 127.5.225.190:39687)
0620 18:43:02.377342 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:02.377348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:02.377568 (+ 220us) server_negotiation.cc:407] Connection header received
0620 18:43:02.377822 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:02.377827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:02.377913 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:02.378025 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:02.378851 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.379569 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.380773 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.381266 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.381397 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:02.386153 (+ 4756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.130:50267: BlockingRecv error: recv got EOF from 127.5.225.130:50267 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Jun 20 18:43:02 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.129@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:43:02 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.129@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:43:02.774745 15293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:02.765092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.129:35541)
0620 18:43:02.765616 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:02.765632 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:02.765739 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:02.766050 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:02.766053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:02.766493 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:02.766699 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:02.766709 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.767914 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.767918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:02.768813 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:02.768824 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.769893 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.769898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:02.770024 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:02.770766 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:02.770807 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:02.774370 (+ 3563us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":258,"threads_started":1}
W20260620 18:43:02.774809 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:02.765162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:35541 (local address 127.5.225.190:39687)
0620 18:43:02.765388 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:02.765393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:02.765718 (+ 325us) server_negotiation.cc:407] Connection header received
0620 18:43:02.765846 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:02.765850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:02.765915 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:02.766012 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:02.766861 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.767786 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.769064 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.769714 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.769864 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:02.774556 (+ 4692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.129:35541: BlockingRecv error: recv got EOF from 127.5.225.129:35541 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Jun 20 18:43:03 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.131@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:43:03 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.131@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:43:03.047605 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:03.037971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.131:36101)
0620 18:43:03.038531 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:03.038546 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:03.038744 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:03.039082 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:03.039086 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:03.039513 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:03.039659 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:03.039675 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:03.040728 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:03.040733 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:03.041616 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:03.041626 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:03.042478 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:03.042483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:03.042656 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:03.043340 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:03.043358 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:03.047282 (+ 3924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":332,"threads_started":1}
W20260620 18:43:03.047701 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:03.038058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:36101 (local address 127.5.225.190:39687)
0620 18:43:03.038404 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:03.038410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:03.038641 (+ 231us) server_negotiation.cc:407] Connection header received
0620 18:43:03.038874 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:03.038880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:03.038950 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:03.039051 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:03.039815 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:03.040570 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:03.041770 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:03.042328 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:03.042447 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:03.047468 (+ 5021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.131:36101: BlockingRecv error: recv got EOF from 127.5.225.131:36101 (error 108)
Metrics: {"server-negotiator.queue_time_us":172}
Jun 20 18:43:03 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.130@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:43:03 dist-test-slave-k6lg krb5kdc[14248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.225.130@KRBTEST.COM for oryx/127.5.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:43:03.398241 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:03.387379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39687 (local address 127.5.225.130:46865)
0620 18:43:03.387830 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:03.387846 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:03.387958 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:03.388277 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:03.388280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:03.388621 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:03.388804 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:03.388813 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:03.390119 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:03.390123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:03.391426 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:03.391438 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:03.392079 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:03.392083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:03.392199 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:03.392854 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:03.392871 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:03.397784 (+ 4913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.225.190:39687: Server oryx/127.5.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":223,"threads_started":1}
W20260620 18:43:03.398391 14985 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:03.387433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:46865 (local address 127.5.225.190:39687)
0620 18:43:03.387654 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:03.387662 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:03.387928 (+ 266us) server_negotiation.cc:407] Connection header received
0620 18:43:03.388052 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:03.388056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:03.388145 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:03.388224 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:03.388945 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:03.389985 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:03.391547 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:03.391962 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:03.392055 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:03.398057 (+ 6002us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.225.130:46865: BlockingRecv error: recv got EOF from 127.5.225.130:46865 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:43:03.761240 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 14835
I20260620 18:43:03.774304 15297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:03.762417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53706 (local address 127.5.225.129:46665)
0620 18:43:03.763065 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:03.763072 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:03.763099 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:43:03.763157 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:03.763163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:03.763426 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:03.763591 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:03.765135 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:03.766180 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:03.767495 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:03.768037 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:03.768134 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:03.770728 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:03.770736 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:03.770751 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:03.770787 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:03.773644 (+ 2857us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.225.129@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 18:43:03.773976 (+ 332us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53706: Request ticket server kudu/127.5.225.129@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":362,"threads_started":1}
W20260620 18:43:03.774410 15297 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53706: Request ticket server kudu/127.5.225.129@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 18:43:03.774447 6022 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 14835: Not authorized: Client connection negotiation failed: client connection to 127.5.225.129:46665: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.225.129@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 18:43:03.774853 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 14835
I20260620 18:43:03.791949 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 14987
I20260620 18:43:03.805078 15298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:03.793127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45012 (local address 127.5.225.130:36925)
0620 18:43:03.793797 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:03.793805 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:03.793835 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:43:03.793897 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:03.793903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:03.794146 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:03.794386 (+ 240us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:03.796084 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:03.797211 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:03.798544 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:03.799129 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:03.799243 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:03.801826 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:03.801835 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:03.801849 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:03.801889 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:03.804573 (+ 2684us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.225.130@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 18:43:03.804847 (+ 274us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45012: Request ticket server kudu/127.5.225.130@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":338,"threads_started":1}
W20260620 18:43:03.805205 15298 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45012: Request ticket server kudu/127.5.225.130@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 18:43:03.805207 6022 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 14987: Not authorized: Client connection negotiation failed: client connection to 127.5.225.130:36925: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.225.130@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 18:43:03.805472 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 14987
I20260620 18:43:03.822098 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 15120
I20260620 18:43:03.837875 15299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:03.823354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54396 (local address 127.5.225.131:39539)
0620 18:43:03.826730 (+ 3376us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:03.826738 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:03.826768 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:43:03.826847 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:03.826856 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:03.827115 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:03.827359 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:03.828783 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:03.829855 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:03.831205 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:03.831770 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:03.831938 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:03.834758 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:03.834767 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:03.834782 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:03.834824 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:03.837322 (+ 2498us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.225.131@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 18:43:03.837615 (+ 293us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54396: Request ticket server kudu/127.5.225.131@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":3142,"thread_start_us":2992,"threads_started":1}
W20260620 18:43:03.838017 15299 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54396: Request ticket server kudu/127.5.225.131@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 18:43:03.837985 6022 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 15120: Not authorized: Client connection negotiation failed: client connection to 127.5.225.131:39539: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.225.131@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 18:43:03.838155 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 15120
I20260620 18:43:03.854854 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 14766
I20260620 18:43:03.856254 14828 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:03.972992 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 14766
2026-06-20T18:43:03Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21435 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T18:43:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:04Z Disabled control of system clock
I20260620 18:43:04.057246 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:33725
--webserver_interface=127.5.225.190
--webserver_port=0
--builtin_ntp_servers=127.5.225.148:40651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:33725
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 18:43:04.377398 15304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:04.377679 15304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:04.377758 15304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:04.384930 15304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:04.385023 15304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:04.385061 15304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:04.385115 15304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:04.385159 15304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:04.394289 15304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:40651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:33725
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:33725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:43:04.395618 15304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:04.397645 15304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:04.408813 15309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:04.409526 15310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:04.410712 15312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:04.411213 15304 server_base.cc:1034] running on GCE node
I20260620 18:43:04.412021 15304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:04.413364 15304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:04.414551 15304 hybrid_clock.cc:648] HybridClock initialized: now 1781980984414490 us; error 52 us; skew 500 ppm
I20260620 18:43:04.414901 15304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:43:04.421409 15304 webserver.cc:460] Webserver started at http://127.5.225.190:39039/ using document root <none> and password file <none>
I20260620 18:43:04.422115 15304 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:04.422271 15304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:04.422591 15304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:04.425405 15304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "7f52848330f74ed392ffc392cb101962"
format_stamp: "Formatted at 2026-06-20 18:43:04 on dist-test-slave-k6lg"
server_key: "cf3bbdcadc27ae2a35c8a8bfc6e5efbe"
server_key_iv: "4aa969257121e83a8bd8b54e49352275"
server_key_version: "encryptionkey@0"
I20260620 18:43:04.426219 15304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "7f52848330f74ed392ffc392cb101962"
format_stamp: "Formatted at 2026-06-20 18:43:04 on dist-test-slave-k6lg"
server_key: "cf3bbdcadc27ae2a35c8a8bfc6e5efbe"
server_key_iv: "4aa969257121e83a8bd8b54e49352275"
server_key_version: "encryptionkey@0"
I20260620 18:43:04.431253 15304 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 18:43:04.434659 15318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:04.436158 15304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:43:04.436357 15304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "7f52848330f74ed392ffc392cb101962"
format_stamp: "Formatted at 2026-06-20 18:43:04 on dist-test-slave-k6lg"
server_key: "cf3bbdcadc27ae2a35c8a8bfc6e5efbe"
server_key_iv: "4aa969257121e83a8bd8b54e49352275"
server_key_version: "encryptionkey@0"
I20260620 18:43:04.436599 15304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:04.451952 15304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:04.452746 15304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:04.453028 15304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:04.470871 15304 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:33725
I20260620 18:43:04.470889 15369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:33725 every 8 connection(s)
I20260620 18:43:04.472198 15304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:04.475850 15370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:04.476271 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 15304
I20260620 18:43:04.476467 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:43:04.477095 6022 external_mini_cluster.cc:1428] Setting key e51197e0f60d84001fe28295eccfc594
I20260620 18:43:04.490662 15370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f52848330f74ed392ffc392cb101962: Bootstrap starting.
I20260620 18:43:04.492916 15373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:04.479176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34314 (local address 127.5.225.190:33725)
0620 18:43:04.480240 (+ 1064us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:04.480263 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:04.480314 (+ 51us) server_negotiation.cc:407] Connection header received
0620 18:43:04.482994 (+ 2680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:04.483022 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:04.483613 (+ 591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:04.484054 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:04.484562 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.486404 (+ 1842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.487717 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.488814 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.488979 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:04.490069 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:04.490079 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:04.490096 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:04.490825 (+ 729us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:04.490919 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:04.490951 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:04.491071 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:04.491302 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 18:43:04.491601 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":321,"threads_started":1}
I20260620 18:43:04.494544 15370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f52848330f74ed392ffc392cb101962: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:04.495602 15370 log.cc:824] T 00000000000000000000000000000000 P 7f52848330f74ed392ffc392cb101962: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:04.498311 15370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f52848330f74ed392ffc392cb101962: No bootstrap required, opened a new log
I20260620 18:43:04.505295 15370 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f52848330f74ed392ffc392cb101962 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f52848330f74ed392ffc392cb101962" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 33725 } }
I20260620 18:43:04.505600 15370 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f52848330f74ed392ffc392cb101962 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:04.505662 15370 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f52848330f74ed392ffc392cb101962 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f52848330f74ed392ffc392cb101962, State: Initialized, Role: FOLLOWER
I20260620 18:43:04.506258 15370 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f52848330f74ed392ffc392cb101962 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f52848330f74ed392ffc392cb101962" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 33725 } }
I20260620 18:43:04.506454 15370 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f52848330f74ed392ffc392cb101962 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:04.506547 15370 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f52848330f74ed392ffc392cb101962 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:04.506669 15370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f52848330f74ed392ffc392cb101962 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:04.508913 15370 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f52848330f74ed392ffc392cb101962 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f52848330f74ed392ffc392cb101962" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 33725 } }
I20260620 18:43:04.509349 15370 leader_election.cc:302] T 00000000000000000000000000000000 P 7f52848330f74ed392ffc392cb101962 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f52848330f74ed392ffc392cb101962; no voters:
I20260620 18:43:04.509895 15370 leader_election.cc:288] T 00000000000000000000000000000000 P 7f52848330f74ed392ffc392cb101962 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:04.510062 15375 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f52848330f74ed392ffc392cb101962 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:04.510783 15375 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f52848330f74ed392ffc392cb101962 [term 1 LEADER]: Becoming Leader. State: Replica: 7f52848330f74ed392ffc392cb101962, State: Running, Role: LEADER
I20260620 18:43:04.511329 15375 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f52848330f74ed392ffc392cb101962 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f52848330f74ed392ffc392cb101962" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 33725 } }
I20260620 18:43:04.512022 15370 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f52848330f74ed392ffc392cb101962 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:04.514283 15376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f52848330f74ed392ffc392cb101962 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f52848330f74ed392ffc392cb101962" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f52848330f74ed392ffc392cb101962" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 33725 } } }
I20260620 18:43:04.514439 15377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f52848330f74ed392ffc392cb101962 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f52848330f74ed392ffc392cb101962. Latest consensus state: current_term: 1 leader_uuid: "7f52848330f74ed392ffc392cb101962" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f52848330f74ed392ffc392cb101962" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 33725 } } }
I20260620 18:43:04.514673 15376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f52848330f74ed392ffc392cb101962 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:04.514695 15377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f52848330f74ed392ffc392cb101962 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:04.516907 15383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:04.522017 15383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:04.530113 15383 catalog_manager.cc:1347] Generated new cluster ID: f77942186c474fba9c04792f543f9dd1
I20260620 18:43:04.530256 15383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:04.544345 15383 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:04.546401 15383 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:04.555074 15383 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f52848330f74ed392ffc392cb101962: Generated new TSK 0
I20260620 18:43:04.555855 15383 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:04.561196 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.129:0
--local_ip_for_outbound_sockets=127.5.225.129
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:33725
--builtin_ntp_servers=127.5.225.148:40651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 18:43:04.891515 15394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:04.891824 15394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:04.891985 15394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:04.899194 15394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:04.899295 15394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:04.899484 15394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.129
I20260620 18:43:04.908592 15394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:40651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:33725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.129
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:43:04.909926 15394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:04.911849 15394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:04.925578 15400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:04.925613 15402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:04.925539 15399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:04.926597 15394 server_base.cc:1034] running on GCE node
I20260620 18:43:04.927268 15394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:04.928248 15394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:04.929467 15394 hybrid_clock.cc:648] HybridClock initialized: now 1781980984929412 us; error 41 us; skew 500 ppm
I20260620 18:43:04.929843 15394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:43:04.932428 15394 webserver.cc:460] Webserver started at http://127.5.225.129:42563/ using document root <none> and password file <none>
I20260620 18:43:04.933137 15394 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:04.933260 15394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:04.933593 15394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:04.936484 15394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/data/instance:
uuid: "acc69d45190a4af79cdbcdc6a17da3df"
format_stamp: "Formatted at 2026-06-20 18:43:04 on dist-test-slave-k6lg"
server_key: "0926e903be9a1313b63be939cfa44233"
server_key_iv: "6ed70f8b418435a2a4be39df5a8f88aa"
server_key_version: "encryptionkey@0"
I20260620 18:43:04.937309 15394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance:
uuid: "acc69d45190a4af79cdbcdc6a17da3df"
format_stamp: "Formatted at 2026-06-20 18:43:04 on dist-test-slave-k6lg"
server_key: "0926e903be9a1313b63be939cfa44233"
server_key_iv: "6ed70f8b418435a2a4be39df5a8f88aa"
server_key_version: "encryptionkey@0"
I20260620 18:43:04.942517 15394 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:43:04.945757 15408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:04.947160 15394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:43:04.947335 15394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/wal
uuid: "acc69d45190a4af79cdbcdc6a17da3df"
format_stamp: "Formatted at 2026-06-20 18:43:04 on dist-test-slave-k6lg"
server_key: "0926e903be9a1313b63be939cfa44233"
server_key_iv: "6ed70f8b418435a2a4be39df5a8f88aa"
server_key_version: "encryptionkey@0"
I20260620 18:43:04.947540 15394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:04.975020 15394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:04.975881 15394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:04.976187 15394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:04.977191 15394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:04.978742 15394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:04.978827 15394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:04.978971 15394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:04.979067 15394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:05.013118 15394 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.129:34487
I20260620 18:43:05.013208 15520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.129:34487 every 8 connection(s)
I20260620 18:43:05.014612 15394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
I20260620 18:43:05.023222 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 15394
I20260620 18:43:05.023432 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance
I20260620 18:43:05.024152 6022 external_mini_cluster.cc:1428] Setting key 230cc32994b039399c11c313e58e6819
I20260620 18:43:05.027149 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.130:0
--local_ip_for_outbound_sockets=127.5.225.130
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:33725
--builtin_ntp_servers=127.5.225.148:40651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 18:43:05.029116 15523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:05.017540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:48847 (local address 127.5.225.190:33725)
0620 18:43:05.018001 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:05.018006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:05.019469 (+ 1463us) server_negotiation.cc:407] Connection header received
0620 18:43:05.021324 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:05.021330 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:05.021426 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:05.021557 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:05.023002 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.023856 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.026001 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.026505 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.026635 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:05.028191 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:05.028199 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:05.028204 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:05.028227 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:05.028268 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:05.028282 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:05.028401 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:05.028646 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 18:43:05.028763 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":233,"threads_started":1}
I20260620 18:43:05.030419 15524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:05.018258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:33725 (local address 127.5.225.129:48847)
0620 18:43:05.019277 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:05.019321 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:05.021009 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:05.021816 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:05.021825 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:05.021927 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:05.022665 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:05.022684 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.024034 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.024039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:05.025847 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:05.025862 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.026758 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.026766 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:05.027005 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:05.027780 (+ 775us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:43:05.027783 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:05.027842 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:43:05.027948 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:05.028400 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:05.028415 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:05.028426 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 18:43:05.028647 (+ 221us) client_negotiation.cc:240] Negotiation successful
0620 18:43:05.029116 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":320,"threads_started":1}
I20260620 18:43:05.032297 15521 heartbeater.cc:347] Connected to a master server at 127.5.225.190:33725
I20260620 18:43:05.032644 15521 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:05.033464 15521 heartbeater.cc:511] Master 127.5.225.190:33725 requested a full tablet report, sending...
I20260620 18:43:05.035831 15335 ts_manager.cc:194] Registered new tserver with Master: acc69d45190a4af79cdbcdc6a17da3df (127.5.225.129:34487)
I20260620 18:43:05.037403 15335 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.225.129:48847
W20260620 18:43:05.349325 15525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:05.349609 15525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:05.349795 15525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:05.357085 15525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:05.357175 15525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:05.357275 15525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.130
I20260620 18:43:05.366256 15525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:40651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:33725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.130
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:43:05.367432 15525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:05.369095 15525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:05.382042 15530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:05.382088 15531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:05.382004 15533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:05.387863 15525 server_base.cc:1034] running on GCE node
I20260620 18:43:05.388495 15525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:05.389438 15525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:05.390667 15525 hybrid_clock.cc:648] HybridClock initialized: now 1781980985390625 us; error 61 us; skew 500 ppm
I20260620 18:43:05.391014 15525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:43:05.393235 15525 webserver.cc:460] Webserver started at http://127.5.225.130:37827/ using document root <none> and password file <none>
I20260620 18:43:05.393980 15525 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:05.394109 15525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:05.394493 15525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:05.397490 15525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/data/instance:
uuid: "e5a89b63d20d4f149dd64809dcbeae43"
format_stamp: "Formatted at 2026-06-20 18:43:05 on dist-test-slave-k6lg"
server_key: "1d6908e55e33ff8c88b9821573e4f650"
server_key_iv: "a866f39bd79eb3f8cb557210e0fb4855"
server_key_version: "encryptionkey@0"
I20260620 18:43:05.398379 15525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance:
uuid: "e5a89b63d20d4f149dd64809dcbeae43"
format_stamp: "Formatted at 2026-06-20 18:43:05 on dist-test-slave-k6lg"
server_key: "1d6908e55e33ff8c88b9821573e4f650"
server_key_iv: "a866f39bd79eb3f8cb557210e0fb4855"
server_key_version: "encryptionkey@0"
I20260620 18:43:05.405323 15525 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260620 18:43:05.408716 15539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:05.410045 15525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:43:05.410280 15525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/wal
uuid: "e5a89b63d20d4f149dd64809dcbeae43"
format_stamp: "Formatted at 2026-06-20 18:43:05 on dist-test-slave-k6lg"
server_key: "1d6908e55e33ff8c88b9821573e4f650"
server_key_iv: "a866f39bd79eb3f8cb557210e0fb4855"
server_key_version: "encryptionkey@0"
I20260620 18:43:05.410501 15525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:05.434417 15525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:05.435305 15525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:05.435616 15525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:05.436614 15525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:05.438069 15525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:05.438208 15525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:05.438354 15525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:05.438441 15525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:05.472275 15525 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.130:43115
I20260620 18:43:05.472309 15651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.130:43115 every 8 connection(s)
I20260620 18:43:05.473747 15525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
I20260620 18:43:05.482309 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 15525
I20260620 18:43:05.482483 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance
I20260620 18:43:05.482956 6022 external_mini_cluster.cc:1428] Setting key 374322cf7419d5a6a293a83f59cedc7a
I20260620 18:43:05.485781 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.131:0
--local_ip_for_outbound_sockets=127.5.225.131
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:33725
--builtin_ntp_servers=127.5.225.148:40651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 18:43:05.487457 15523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:05.476690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:53987 (local address 127.5.225.190:33725)
0620 18:43:05.476860 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:05.476865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:05.478401 (+ 1536us) server_negotiation.cc:407] Connection header received
0620 18:43:05.480140 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:05.480146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:05.480244 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:05.480355 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:05.481381 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.482359 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.484686 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.485105 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.485260 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:05.486641 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:05.486647 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:05.486651 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:05.486668 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:05.486697 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:05.486708 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:05.486817 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:05.487048 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 18:43:05.487144 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 18:43:05.488660 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:05.477269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:33725 (local address 127.5.225.130:53987)
0620 18:43:05.478248 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:05.478290 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:05.479881 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:05.480570 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:05.480578 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:05.480670 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:05.481223 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:05.481239 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.482561 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.482567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:05.484538 (+ 1971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:05.484552 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.485347 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.485354 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:05.485585 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:05.486323 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:43:05.486326 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:05.486365 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:43:05.486444 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:05.486807 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:05.486817 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:05.486825 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 18:43:05.487056 (+ 231us) client_negotiation.cc:240] Negotiation successful
0620 18:43:05.487491 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":289,"threads_started":1}
I20260620 18:43:05.490051 15652 heartbeater.cc:347] Connected to a master server at 127.5.225.190:33725
I20260620 18:43:05.490376 15652 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:05.491096 15652 heartbeater.cc:511] Master 127.5.225.190:33725 requested a full tablet report, sending...
I20260620 18:43:05.492400 15335 ts_manager.cc:194] Registered new tserver with Master: e5a89b63d20d4f149dd64809dcbeae43 (127.5.225.130:43115)
I20260620 18:43:05.493192 15335 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.225.130:53987
W20260620 18:43:05.809116 15655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:05.809372 15655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:05.809510 15655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:05.817032 15655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:05.817145 15655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:05.817335 15655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.131
I20260620 18:43:05.826416 15655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:40651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:33725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.131
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:43:05.827728 15655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:05.829568 15655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:05.842794 15661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:05.842855 15663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:05.842785 15660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:05.843806 15655 server_base.cc:1034] running on GCE node
I20260620 18:43:05.844389 15655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:05.845366 15655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:05.846594 15655 hybrid_clock.cc:648] HybridClock initialized: now 1781980985846532 us; error 63 us; skew 500 ppm
I20260620 18:43:05.846933 15655 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:43:05.849174 15655 webserver.cc:460] Webserver started at http://127.5.225.131:38371/ using document root <none> and password file <none>
I20260620 18:43:05.849907 15655 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:05.850041 15655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:05.850423 15655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:05.853365 15655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/data/instance:
uuid: "4614ce77f6a2493aa4516fdf0bf5b5d5"
format_stamp: "Formatted at 2026-06-20 18:43:05 on dist-test-slave-k6lg"
server_key: "669f1e1deb0868fbf65de9b42a4b56b5"
server_key_iv: "3b969d2ffbae65bdf65fc314b37643a0"
server_key_version: "encryptionkey@0"
I20260620 18:43:05.854207 15655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance:
uuid: "4614ce77f6a2493aa4516fdf0bf5b5d5"
format_stamp: "Formatted at 2026-06-20 18:43:05 on dist-test-slave-k6lg"
server_key: "669f1e1deb0868fbf65de9b42a4b56b5"
server_key_iv: "3b969d2ffbae65bdf65fc314b37643a0"
server_key_version: "encryptionkey@0"
I20260620 18:43:05.859333 15655 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:43:05.862820 15669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:05.864132 15655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:43:05.864300 15655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/wal
uuid: "4614ce77f6a2493aa4516fdf0bf5b5d5"
format_stamp: "Formatted at 2026-06-20 18:43:05 on dist-test-slave-k6lg"
server_key: "669f1e1deb0868fbf65de9b42a4b56b5"
server_key_iv: "3b969d2ffbae65bdf65fc314b37643a0"
server_key_version: "encryptionkey@0"
I20260620 18:43:05.864503 15655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:05.884887 15655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:05.885721 15655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:05.886040 15655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:05.887002 15655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:05.888381 15655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:05.888500 15655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:05.888628 15655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:05.888713 15655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:05.922405 15655 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.131:45943
I20260620 18:43:05.922487 15781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.131:45943 every 8 connection(s)
I20260620 18:43:05.923703 15655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
I20260620 18:43:05.929206 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 15655
I20260620 18:43:05.929376 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance
I20260620 18:43:05.929828 6022 external_mini_cluster.cc:1428] Setting key 4cb53437c12242d1dc77c39e00617c9f
I20260620 18:43:05.936298 15523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:05.926443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:41789 (local address 127.5.225.190:33725)
0620 18:43:05.926643 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:05.926650 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:05.928036 (+ 1386us) server_negotiation.cc:407] Connection header received
0620 18:43:05.929951 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:05.929959 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:05.930054 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:05.930165 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:05.931350 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.932091 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.933597 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.934014 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.934143 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:05.935519 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:05.935524 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:05.935527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:05.935543 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:05.935571 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:05.935582 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:05.935676 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:05.935893 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 18:43:05.935982 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:43:05.937274 15784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:05.926979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:33725 (local address 127.5.225.131:41789)
0620 18:43:05.927865 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:05.927901 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:05.929663 (+ 1762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:05.930416 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:05.930424 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:05.930512 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:05.931183 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:05.931202 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.932251 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.932255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:05.933457 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:05.933468 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.934268 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.934277 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:05.934493 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:05.935202 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:43:05.935205 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:05.935250 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:43:05.935328 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:05.935675 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:05.935687 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:05.935697 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 18:43:05.935869 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 18:43:05.936259 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":266,"threads_started":1}
I20260620 18:43:05.938534 15782 heartbeater.cc:347] Connected to a master server at 127.5.225.190:33725
I20260620 18:43:05.938814 15782 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:05.939553 15782 heartbeater.cc:511] Master 127.5.225.190:33725 requested a full tablet report, sending...
I20260620 18:43:05.940909 15335 ts_manager.cc:194] Registered new tserver with Master: 4614ce77f6a2493aa4516fdf0bf5b5d5 (127.5.225.131:45943)
I20260620 18:43:05.941625 15335 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.225.131:41789
I20260620 18:43:05.944759 6022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 18:43:05.950495 15523 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:05.948985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34330 (local address 127.5.225.190:33725)
0620 18:43:05.949169 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:05.949173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:05.949681 (+ 508us) server_negotiation.cc:407] Connection header received
0620 18:43:05.949798 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:05.949802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:05.949869 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:05.949948 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:05.950333 (+ 385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34330: BlockingRecv error: recv got EOF from 127.0.0.1:34330 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 18:43:05.951632 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 15394
I20260620 18:43:05.958899 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:05.952468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59618 (local address 127.5.225.129:34487)
0620 18:43:05.952951 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:05.952957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:05.952981 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:43:05.953032 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:05.953037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:05.953214 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:05.953336 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:05.953648 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.954707 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.955867 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.956403 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.956501 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:05.957446 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:05.957457 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:05.957470 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:05.958066 (+ 596us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:05.958108 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:05.958126 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:05.958318 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:05.958435 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 18:43:05.958586 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":280,"threads_started":1}
I20260620 18:43:05.960009 15516 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:06.040951 15521 heartbeater.cc:503] Master 127.5.225.190:33725 was elected leader, sending a full tablet report...
I20260620 18:43:06.042187 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 15394
W20260620 18:43:06.059002 15322 connection.cc:535] server connection from 127.5.225.129:48847 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 18:43:06.059338 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 15525
I20260620 18:43:06.067795 15793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:06.060281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43636 (local address 127.5.225.130:43115)
0620 18:43:06.060885 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:06.060893 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:06.060921 (+ 28us) server_negotiation.cc:407] Connection header received
0620 18:43:06.060980 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:06.060987 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:06.061229 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:06.061358 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:06.061864 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.062962 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.064248 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.064817 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.064961 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:06.066147 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:06.066202 (+ 55us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:06.066221 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:06.066855 (+ 634us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:06.066902 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:06.066923 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:06.067052 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:06.067234 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 18:43:06.067363 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":324,"threads_started":1}
I20260620 18:43:06.069140 15647 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:06.149369 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 15525
I20260620 18:43:06.165706 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 15655
I20260620 18:43:06.173806 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:06.166702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39722 (local address 127.5.225.131:45943)
0620 18:43:06.167331 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:06.167337 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:06.167367 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:43:06.167424 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:06.167430 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:06.167692 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:06.167820 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:06.168277 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.169475 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.170913 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.171458 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.171569 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:06.172485 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:06.172492 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:06.172505 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:06.173123 (+ 618us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:06.173167 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:06.173186 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:06.173274 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:06.173395 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 18:43:06.173493 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":340,"threads_started":1}
I20260620 18:43:06.175091 15777 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:06.256341 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 15655
I20260620 18:43:06.272037 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 15304
I20260620 18:43:06.273342 15365 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:06.377218 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 15304
2026-06-20T18:43:06Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2389 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T18:43:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:06Z Disabled control of system clock
I20260620 18:43:06.446916 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:43729
--webserver_interface=127.5.225.190
--webserver_port=0
--builtin_ntp_servers=127.5.225.148:33929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:43729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 18:43:06.763383 15801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:06.763656 15801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:06.763787 15801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:06.771042 15801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:06.771135 15801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:06.771220 15801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:06.771325 15801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:06.771404 15801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:06.780298 15801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:33929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:43729
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:43729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:43:06.781525 15801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:06.783259 15801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:06.795452 15806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:06.795368 15809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:06.795420 15807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:06.796401 15801 server_base.cc:1034] running on GCE node
I20260620 18:43:06.797144 15801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:06.798504 15801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:06.799727 15801 hybrid_clock.cc:648] HybridClock initialized: now 1781980986799693 us; error 49 us; skew 500 ppm
I20260620 18:43:06.800073 15801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:43:06.802248 15801 webserver.cc:460] Webserver started at http://127.5.225.190:43453/ using document root <none> and password file <none>
I20260620 18:43:06.802909 15801 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:06.803037 15801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:06.803345 15801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:06.806142 15801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "95771fe327104770b92e3b1e5fdd384d"
format_stamp: "Formatted at 2026-06-20 18:43:06 on dist-test-slave-k6lg"
server_key: "1cfe4c9f061d6152f42a72e8c92df999"
server_key_iv: "14c3774aa690d3b268a9131fffdbdcbc"
server_key_version: "encryptionkey@0"
I20260620 18:43:06.807014 15801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "95771fe327104770b92e3b1e5fdd384d"
format_stamp: "Formatted at 2026-06-20 18:43:06 on dist-test-slave-k6lg"
server_key: "1cfe4c9f061d6152f42a72e8c92df999"
server_key_iv: "14c3774aa690d3b268a9131fffdbdcbc"
server_key_version: "encryptionkey@0"
I20260620 18:43:06.811828 15801 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 18:43:06.815006 15815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:06.816231 15801 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:43:06.816408 15801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "95771fe327104770b92e3b1e5fdd384d"
format_stamp: "Formatted at 2026-06-20 18:43:06 on dist-test-slave-k6lg"
server_key: "1cfe4c9f061d6152f42a72e8c92df999"
server_key_iv: "14c3774aa690d3b268a9131fffdbdcbc"
server_key_version: "encryptionkey@0"
I20260620 18:43:06.816618 15801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:06.822041 15801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:06.822788 15801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:06.823088 15801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:06.841486 15801 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:43729
I20260620 18:43:06.841506 15866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:43729 every 8 connection(s)
I20260620 18:43:06.842953 15801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:06.846452 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 15801
I20260620 18:43:06.846659 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:43:06.846805 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:06.847323 6022 external_mini_cluster.cc:1428] Setting key 36d466b52c374b78de0058c2e307d3b3
I20260620 18:43:06.856561 15870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:06.849286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35042 (local address 127.5.225.190:43729)
0620 18:43:06.850460 (+ 1174us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:06.850475 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:06.850521 (+ 46us) server_negotiation.cc:407] Connection header received
0620 18:43:06.852717 (+ 2196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:06.852744 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:06.853265 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:06.853659 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:06.854869 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:06.854876 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:06.854891 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:06.854907 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:06.854979 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:06.854996 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:06.855192 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:06.855294 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 18:43:06.855601 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":437,"threads_started":1}
I20260620 18:43:06.860548 15867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95771fe327104770b92e3b1e5fdd384d: Bootstrap starting.
I20260620 18:43:06.863765 15867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95771fe327104770b92e3b1e5fdd384d: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:06.864763 15867 log.cc:824] T 00000000000000000000000000000000 P 95771fe327104770b92e3b1e5fdd384d: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:06.868014 15867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95771fe327104770b92e3b1e5fdd384d: No bootstrap required, opened a new log
I20260620 18:43:06.875691 15867 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95771fe327104770b92e3b1e5fdd384d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95771fe327104770b92e3b1e5fdd384d" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 43729 } }
I20260620 18:43:06.876145 15867 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95771fe327104770b92e3b1e5fdd384d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:06.876263 15867 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95771fe327104770b92e3b1e5fdd384d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95771fe327104770b92e3b1e5fdd384d, State: Initialized, Role: FOLLOWER
I20260620 18:43:06.876868 15867 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95771fe327104770b92e3b1e5fdd384d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95771fe327104770b92e3b1e5fdd384d" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 43729 } }
I20260620 18:43:06.877113 15867 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95771fe327104770b92e3b1e5fdd384d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:06.877259 15867 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95771fe327104770b92e3b1e5fdd384d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:06.877434 15867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95771fe327104770b92e3b1e5fdd384d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:06.879703 15867 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95771fe327104770b92e3b1e5fdd384d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95771fe327104770b92e3b1e5fdd384d" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 43729 } }
I20260620 18:43:06.880215 15867 leader_election.cc:302] T 00000000000000000000000000000000 P 95771fe327104770b92e3b1e5fdd384d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95771fe327104770b92e3b1e5fdd384d; no voters:
I20260620 18:43:06.880869 15867 leader_election.cc:288] T 00000000000000000000000000000000 P 95771fe327104770b92e3b1e5fdd384d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:06.881050 15872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95771fe327104770b92e3b1e5fdd384d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:06.881824 15872 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95771fe327104770b92e3b1e5fdd384d [term 1 LEADER]: Becoming Leader. State: Replica: 95771fe327104770b92e3b1e5fdd384d, State: Running, Role: LEADER
I20260620 18:43:06.882474 15872 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95771fe327104770b92e3b1e5fdd384d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95771fe327104770b92e3b1e5fdd384d" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 43729 } }
I20260620 18:43:06.883307 15867 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95771fe327104770b92e3b1e5fdd384d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:06.885603 15874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95771fe327104770b92e3b1e5fdd384d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95771fe327104770b92e3b1e5fdd384d. Latest consensus state: current_term: 1 leader_uuid: "95771fe327104770b92e3b1e5fdd384d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95771fe327104770b92e3b1e5fdd384d" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 43729 } } }
I20260620 18:43:06.885984 15874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95771fe327104770b92e3b1e5fdd384d [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:06.885686 15873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95771fe327104770b92e3b1e5fdd384d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95771fe327104770b92e3b1e5fdd384d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95771fe327104770b92e3b1e5fdd384d" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 43729 } } }
I20260620 18:43:06.886121 15873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95771fe327104770b92e3b1e5fdd384d [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:06.888294 15880 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:06.893303 15880 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:06.901571 15880 catalog_manager.cc:1347] Generated new cluster ID: 3d46687e33fa4c2488d3fa5b2941d396
I20260620 18:43:06.901685 15880 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:06.937885 15880 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:06.938431 15880 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:06.945003 15880 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95771fe327104770b92e3b1e5fdd384d: Generated new TSK 0
I20260620 18:43:06.946035 15880 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:06.958148 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.129:0
--local_ip_for_outbound_sockets=127.5.225.129
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:43729
--builtin_ntp_servers=127.5.225.148:33929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 18:43:07.298866 15891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:07.299151 15891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:07.299247 15891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:07.306530 15891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:07.306640 15891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:07.306754 15891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.129
I20260620 18:43:07.315966 15891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:33929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:43729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.129
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:43:07.317314 15891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:07.319191 15891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:07.332970 15896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:07.332996 15899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:07.332940 15897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:07.334018 15891 server_base.cc:1034] running on GCE node
I20260620 18:43:07.334654 15891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:07.335685 15891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:07.336932 15891 hybrid_clock.cc:648] HybridClock initialized: now 1781980987336880 us; error 78 us; skew 500 ppm
I20260620 18:43:07.337263 15891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:43:07.339689 15891 webserver.cc:460] Webserver started at http://127.5.225.129:36271/ using document root <none> and password file <none>
I20260620 18:43:07.340397 15891 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:07.340529 15891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:07.340866 15891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:07.343768 15891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/data/instance:
uuid: "4f3e55fb943a438c9c8b8579a60274e0"
format_stamp: "Formatted at 2026-06-20 18:43:07 on dist-test-slave-k6lg"
server_key: "5234de6af5096caa39cbf7825bb548ec"
server_key_iv: "7ac2c794b623756fb5017a15bf8ddad1"
server_key_version: "encryptionkey@0"
I20260620 18:43:07.344578 15891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance:
uuid: "4f3e55fb943a438c9c8b8579a60274e0"
format_stamp: "Formatted at 2026-06-20 18:43:07 on dist-test-slave-k6lg"
server_key: "5234de6af5096caa39cbf7825bb548ec"
server_key_iv: "7ac2c794b623756fb5017a15bf8ddad1"
server_key_version: "encryptionkey@0"
I20260620 18:43:07.349923 15891 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:43:07.353399 15905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:07.354734 15891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:43:07.354918 15891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/wal
uuid: "4f3e55fb943a438c9c8b8579a60274e0"
format_stamp: "Formatted at 2026-06-20 18:43:07 on dist-test-slave-k6lg"
server_key: "5234de6af5096caa39cbf7825bb548ec"
server_key_iv: "7ac2c794b623756fb5017a15bf8ddad1"
server_key_version: "encryptionkey@0"
I20260620 18:43:07.355142 15891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:07.360564 15891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:07.361321 15891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:07.361591 15891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:07.362480 15891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:07.363814 15891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:07.363940 15891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:07.364073 15891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:07.364161 15891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:07.405548 15891 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.129:44427
I20260620 18:43:07.405591 16017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.129:44427 every 8 connection(s)
I20260620 18:43:07.407135 15891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
I20260620 18:43:07.410950 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 15891
I20260620 18:43:07.411171 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance
I20260620 18:43:07.412014 6022 external_mini_cluster.cc:1428] Setting key 781ef440df23468013e1dda8719f62c6
I20260620 18:43:07.414680 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.130:0
--local_ip_for_outbound_sockets=127.5.225.130
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:43729
--builtin_ntp_servers=127.5.225.148:33929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 18:43:07.416363 16020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.410240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:57287 (local address 127.5.225.190:43729)
0620 18:43:07.410691 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:07.410732 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:07.412097 (+ 1365us) server_negotiation.cc:407] Connection header received
0620 18:43:07.413983 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:07.413989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:07.414054 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:07.414213 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:07.415565 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:07.415571 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:07.415576 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:07.415595 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:07.415646 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:07.415658 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:07.415788 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:07.416014 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 18:43:07.416078 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":235,"threads_started":1}
I20260620 18:43:07.417330 16021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.410929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:43729 (local address 127.5.225.129:57287)
0620 18:43:07.411930 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:07.411975 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:07.413668 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:07.414430 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:07.414438 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:07.414532 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:07.415274 (+ 742us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:43:07.415276 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:07.415315 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:43:07.415399 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:07.415768 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:07.415779 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:07.415786 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 18:43:07.416070 (+ 284us) client_negotiation.cc:240] Negotiation successful
0620 18:43:07.416316 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":322,"threads_started":1}
I20260620 18:43:07.418805 16018 heartbeater.cc:347] Connected to a master server at 127.5.225.190:43729
I20260620 18:43:07.419111 16018 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:07.419807 16018 heartbeater.cc:511] Master 127.5.225.190:43729 requested a full tablet report, sending...
I20260620 18:43:07.421864 15832 ts_manager.cc:194] Registered new tserver with Master: 4f3e55fb943a438c9c8b8579a60274e0 (127.5.225.129:44427)
W20260620 18:43:07.736568 16022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:07.736853 16022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:07.736941 16022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:07.744051 16022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:07.744148 16022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:07.744256 16022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.130
I20260620 18:43:07.753238 16022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:33929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:43729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.130
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:43:07.754495 16022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:07.756211 16022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:07.769488 16027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:07.769414 16028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:07.769510 16030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:07.770197 16022 server_base.cc:1034] running on GCE node
I20260620 18:43:07.770872 16022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:07.771911 16022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:07.773175 16022 hybrid_clock.cc:648] HybridClock initialized: now 1781980987773115 us; error 45 us; skew 500 ppm
I20260620 18:43:07.773535 16022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:43:07.775964 16022 webserver.cc:460] Webserver started at http://127.5.225.130:36405/ using document root <none> and password file <none>
I20260620 18:43:07.776660 16022 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:07.776791 16022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:07.777117 16022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:07.779932 16022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/data/instance:
uuid: "3d9b755ca4ee46bca3a698189de4f266"
format_stamp: "Formatted at 2026-06-20 18:43:07 on dist-test-slave-k6lg"
server_key: "2bd0c22f1db01ccfe733019cfcb6073b"
server_key_iv: "587eb1c7aa2f33a8e456ce8d1d1731a9"
server_key_version: "encryptionkey@0"
I20260620 18:43:07.780727 16022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance:
uuid: "3d9b755ca4ee46bca3a698189de4f266"
format_stamp: "Formatted at 2026-06-20 18:43:07 on dist-test-slave-k6lg"
server_key: "2bd0c22f1db01ccfe733019cfcb6073b"
server_key_iv: "587eb1c7aa2f33a8e456ce8d1d1731a9"
server_key_version: "encryptionkey@0"
I20260620 18:43:07.785708 16022 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:43:07.789052 16036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:07.790416 16022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 18:43:07.790570 16022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/wal
uuid: "3d9b755ca4ee46bca3a698189de4f266"
format_stamp: "Formatted at 2026-06-20 18:43:07 on dist-test-slave-k6lg"
server_key: "2bd0c22f1db01ccfe733019cfcb6073b"
server_key_iv: "587eb1c7aa2f33a8e456ce8d1d1731a9"
server_key_version: "encryptionkey@0"
I20260620 18:43:07.790784 16022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:07.796142 16022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:07.796870 16022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:07.797135 16022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:07.797981 16022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:07.799340 16022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:07.799456 16022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:07.799588 16022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:07.799681 16022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:07.832343 16022 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.130:40333
I20260620 18:43:07.832430 16148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.130:40333 every 8 connection(s)
I20260620 18:43:07.833658 16022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
I20260620 18:43:07.836570 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 16022
I20260620 18:43:07.836855 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance
I20260620 18:43:07.837467 6022 external_mini_cluster.cc:1428] Setting key 01fae805379a36e5cd192bb6d69c2d11
I20260620 18:43:07.840842 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.131:0
--local_ip_for_outbound_sockets=127.5.225.131
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:43729
--builtin_ntp_servers=127.5.225.148:33929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 18:43:07.843881 16020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.836442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:55409 (local address 127.5.225.190:43729)
0620 18:43:07.836678 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:07.836685 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:07.838352 (+ 1667us) server_negotiation.cc:407] Connection header received
0620 18:43:07.840792 (+ 2440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:07.840797 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:07.840897 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:07.841013 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:07.842932 (+ 1919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:07.842941 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:07.842951 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:07.842977 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:07.843027 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:07.843048 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:07.843191 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:07.843428 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 18:43:07.843501 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 18:43:07.844846 16151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.836934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:43729 (local address 127.5.225.130:55409)
0620 18:43:07.838111 (+ 1177us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:07.838221 (+ 110us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:07.840460 (+ 2239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:07.841416 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:07.841432 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:07.841559 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:07.842594 (+ 1035us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:43:07.842597 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:07.842649 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:43:07.842759 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:07.843181 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:07.843191 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:07.843199 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 18:43:07.843481 (+ 282us) client_negotiation.cc:240] Negotiation successful
0620 18:43:07.843725 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":303,"threads_started":1}
I20260620 18:43:07.846321 16149 heartbeater.cc:347] Connected to a master server at 127.5.225.190:43729
I20260620 18:43:07.846665 16149 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:07.847425 16149 heartbeater.cc:511] Master 127.5.225.190:43729 requested a full tablet report, sending...
I20260620 18:43:07.849149 15832 ts_manager.cc:194] Registered new tserver with Master: 3d9b755ca4ee46bca3a698189de4f266 (127.5.225.130:40333)
W20260620 18:43:08.174943 16152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:08.175251 16152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:08.175419 16152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:08.182742 16152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:08.182862 16152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:08.183045 16152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.131
I20260620 18:43:08.193063 16152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:33929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:43729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.131
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:43:08.194530 16152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:08.196328 16152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:08.209962 16158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:08.210007 16157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:08.209930 16160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:08.215612 16152 server_base.cc:1034] running on GCE node
I20260620 18:43:08.216421 16152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:08.217486 16152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:08.218716 16152 hybrid_clock.cc:648] HybridClock initialized: now 1781980988218658 us; error 88 us; skew 500 ppm
I20260620 18:43:08.219058 16152 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:43:08.221599 16152 webserver.cc:460] Webserver started at http://127.5.225.131:33563/ using document root <none> and password file <none>
I20260620 18:43:08.222396 16152 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:08.222530 16152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:08.222872 16152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:08.225756 16152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/data/instance:
uuid: "a13e396144654ddaa9180912cbcafea4"
format_stamp: "Formatted at 2026-06-20 18:43:08 on dist-test-slave-k6lg"
server_key: "46a00d6befd7a850767c4256eaa80d5b"
server_key_iv: "0d393540b2c5b4a073164a141e7fda82"
server_key_version: "encryptionkey@0"
I20260620 18:43:08.226570 16152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance:
uuid: "a13e396144654ddaa9180912cbcafea4"
format_stamp: "Formatted at 2026-06-20 18:43:08 on dist-test-slave-k6lg"
server_key: "46a00d6befd7a850767c4256eaa80d5b"
server_key_iv: "0d393540b2c5b4a073164a141e7fda82"
server_key_version: "encryptionkey@0"
I20260620 18:43:08.231660 16152 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:43:08.235136 16166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:08.236471 16152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:43:08.236651 16152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/wal
uuid: "a13e396144654ddaa9180912cbcafea4"
format_stamp: "Formatted at 2026-06-20 18:43:08 on dist-test-slave-k6lg"
server_key: "46a00d6befd7a850767c4256eaa80d5b"
server_key_iv: "0d393540b2c5b4a073164a141e7fda82"
server_key_version: "encryptionkey@0"
I20260620 18:43:08.236892 16152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:08.242503 16152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:08.243263 16152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:08.243525 16152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:08.244544 16152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:08.245913 16152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:08.246038 16152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:08.246193 16152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:08.246296 16152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 18:43:08.279194 16152 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.131:42493
I20260620 18:43:08.279247 16278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.131:42493 every 8 connection(s)
I20260620 18:43:08.280576 16152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
I20260620 18:43:08.285223 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 16152
I20260620 18:43:08.285420 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance
I20260620 18:43:08.285969 6022 external_mini_cluster.cc:1428] Setting key 6c8a2741c5fd827a5c56687cc0822771
I20260620 18:43:08.289901 16020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:08.283167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:60279 (local address 127.5.225.190:43729)
0620 18:43:08.283352 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:08.283356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:08.284989 (+ 1633us) server_negotiation.cc:407] Connection header received
0620 18:43:08.287281 (+ 2292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:08.287289 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:08.287372 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:08.287526 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:08.289152 (+ 1626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:08.289157 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:08.289161 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:08.289181 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:08.289219 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:08.289235 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:08.289364 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:08.289602 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 18:43:08.289652 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:43:08.290959 16281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:08.283831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:43729 (local address 127.5.225.131:60279)
0620 18:43:08.284753 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:08.284835 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:08.286988 (+ 2153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:08.287821 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:08.287833 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:08.287980 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:08.288819 (+ 839us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:43:08.288822 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:08.288861 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:43:08.288950 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:08.289368 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:08.289380 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:08.289390 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 18:43:08.289668 (+ 278us) client_negotiation.cc:240] Negotiation successful
0620 18:43:08.289872 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":289,"threads_started":1}
I20260620 18:43:08.292169 16279 heartbeater.cc:347] Connected to a master server at 127.5.225.190:43729
I20260620 18:43:08.292476 16279 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:08.293064 16279 heartbeater.cc:511] Master 127.5.225.190:43729 requested a full tablet report, sending...
I20260620 18:43:08.294559 15832 ts_manager.cc:194] Registered new tserver with Master: a13e396144654ddaa9180912cbcafea4 (127.5.225.131:42493)
I20260620 18:43:08.301673 6022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 18:43:08.307679 16020 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:08.306259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35058 (local address 127.5.225.190:43729)
0620 18:43:08.306416 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:08.306421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:08.306917 (+ 496us) server_negotiation.cc:407] Connection header received
0620 18:43:08.307053 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:08.307057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:08.307107 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:08.307208 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:08.307475 (+ 267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35058: BlockingRecv error: recv got EOF from 127.0.0.1:35058 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 18:43:08.308607 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 15891
I20260620 18:43:08.312063 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:08.309606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47814 (local address 127.5.225.129:44427)
0620 18:43:08.310092 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:08.310107 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:08.310133 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:43:08.310242 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:08.310256 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:08.310502 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:08.310631 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:08.311505 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:08.311511 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:08.311523 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:08.311539 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:08.311577 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:08.311590 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:08.311688 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:08.311769 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 18:43:08.311845 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":235,"threads_started":1}
I20260620 18:43:08.313273 16013 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:08.394695 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 15891
I20260620 18:43:08.411036 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 16022
I20260620 18:43:08.414705 16290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:08.412070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43414 (local address 127.5.225.130:40333)
0620 18:43:08.412662 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:08.412671 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:08.412704 (+ 33us) server_negotiation.cc:407] Connection header received
0620 18:43:08.412785 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:08.412794 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:08.413040 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:08.413171 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:08.414055 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:08.414061 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:08.414073 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:08.414089 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:08.414128 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:08.414141 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:08.414315 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:08.414394 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 18:43:08.414474 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":306,"threads_started":1}
I20260620 18:43:08.415903 16144 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:08.496770 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 16022
I20260620 18:43:08.512948 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 16152
I20260620 18:43:08.516561 16292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:08.513876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50096 (local address 127.5.225.131:42493)
0620 18:43:08.514508 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:08.514522 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:08.514554 (+ 32us) server_negotiation.cc:407] Connection header received
0620 18:43:08.514615 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:08.514622 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:08.514885 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:08.515014 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:08.515854 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:08.515861 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:08.515872 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:08.515883 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:08.515923 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:08.515939 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:08.516066 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:08.516112 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 18:43:08.516229 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":323,"threads_started":1}
I20260620 18:43:08.517762 16274 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:08.597922 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 16152
I20260620 18:43:08.614751 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 15801
I20260620 18:43:08.615953 15862 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:08.716725 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 15801
2026-06-20T18:43:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2339 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:08 dist-test-slave-k6lg krb5kdc[16297](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:08 dist-test-slave-k6lg krb5kdc[16297](info): set up 2 sockets
Jun 20 18:43:08 dist-test-slave-k6lg krb5kdc[16297](info): commencing operation
krb5kdc: starting...
W20260620 18:43:09.811304 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:43:09 dist-test-slave-k6lg krb5kdc[16297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
I20260620 18:43:10.046806 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:36783
--webserver_interface=127.5.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:36783
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:10.369745 16310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:10.370057 16310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:10.370242 16310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:10.377720 16310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:10.377839 16310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:10.377949 16310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:10.378062 16310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:10.378146 16310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:10.388198 16310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:36783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:36783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:43:10.389565 16310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:10.390969 16310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:10.402709 16315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:10.402838 16318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:10.402873 16316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:10.404207 16310 server_base.cc:1034] running on GCE node
I20260620 18:43:10.405007 16310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:10.406543 16310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:10.407815 16310 hybrid_clock.cc:648] HybridClock initialized: now 1781980990407758 us; error 33 us; skew 500 ppm
I20260620 18:43:10.408185 16310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:43:10 dist-test-slave-k6lg krb5kdc[16297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980990, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:10.416165 16310 init.cc:377] Logged in from keytab as kudu/127.5.225.190@KRBTEST.COM (short username kudu)
I20260620 18:43:10.418009 16310 webserver.cc:460] Webserver started at http://127.5.225.190:37871/ using document root <none> and password file <none>
I20260620 18:43:10.418977 16310 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:10.419121 16310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:10.419420 16310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:10.422657 16310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "00c6acc5246943af8b315788260205cb"
format_stamp: "Formatted at 2026-06-20 18:43:10 on dist-test-slave-k6lg"
server_key: "382e096c3bca81861b253fa84cee1aab"
server_key_iv: "82d292c22670fe3ccbf262605ae5266f"
server_key_version: "encryptionkey@0"
I20260620 18:43:10.423424 16310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "00c6acc5246943af8b315788260205cb"
format_stamp: "Formatted at 2026-06-20 18:43:10 on dist-test-slave-k6lg"
server_key: "382e096c3bca81861b253fa84cee1aab"
server_key_iv: "82d292c22670fe3ccbf262605ae5266f"
server_key_version: "encryptionkey@0"
I20260620 18:43:10.428716 16310 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 18:43:10.432639 16325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:10.434105 16310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 18:43:10.434388 16310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "00c6acc5246943af8b315788260205cb"
format_stamp: "Formatted at 2026-06-20 18:43:10 on dist-test-slave-k6lg"
server_key: "382e096c3bca81861b253fa84cee1aab"
server_key_iv: "82d292c22670fe3ccbf262605ae5266f"
server_key_version: "encryptionkey@0"
I20260620 18:43:10.434600 16310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:10.447321 16310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:10.451725 16310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:10.452013 16310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:10.470573 16310 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:36783
I20260620 18:43:10.470597 16376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:36783 every 8 connection(s)
I20260620 18:43:10.472101 16310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:10.476065 16377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:10.479398 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 16310
I20260620 18:43:10.479636 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:43:10.480422 6022 external_mini_cluster.cc:1428] Setting key 1204234611e0abac310f158266c43081
I20260620 18:43:10.489169 16377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00c6acc5246943af8b315788260205cb: Bootstrap starting.
I20260620 18:43:10.492983 16377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00c6acc5246943af8b315788260205cb: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:10.494295 16377 log.cc:824] T 00000000000000000000000000000000 P 00c6acc5246943af8b315788260205cb: Log is configured to *not* fsync() on all Append() calls
Jun 20 18:43:10 dist-test-slave-k6lg krb5kdc[16297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:43:10.497254 16377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00c6acc5246943af8b315788260205cb: No bootstrap required, opened a new log
I20260620 18:43:10.503824 16380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:10.482564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58026 (local address 127.5.225.190:36783)
0620 18:43:10.483337 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:10.483350 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:10.483392 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:43:10.485016 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:10.485038 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:10.485442 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:10.485787 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:10.487661 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:10.488957 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:10.490775 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:10.491477 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:10.491610 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:10.496435 (+ 4825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:10.496443 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:10.496458 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:10.496491 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:10.499685 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:10.500408 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:10.500415 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:10.500425 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:10.500594 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:10.500995 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:10.500999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:10.501001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:10.501557 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:10.501889 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:10.502366 (+ 477us) server_negotiation.cc:299] Negotiation successful
0620 18:43:10.502782 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":365,"threads_started":1}
I20260620 18:43:10.505028 16377 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00c6acc5246943af8b315788260205cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00c6acc5246943af8b315788260205cb" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36783 } }
I20260620 18:43:10.505424 16377 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00c6acc5246943af8b315788260205cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:10.505553 16377 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00c6acc5246943af8b315788260205cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c6acc5246943af8b315788260205cb, State: Initialized, Role: FOLLOWER
I20260620 18:43:10.506254 16377 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00c6acc5246943af8b315788260205cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00c6acc5246943af8b315788260205cb" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36783 } }
I20260620 18:43:10.506511 16377 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00c6acc5246943af8b315788260205cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:10.506685 16377 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00c6acc5246943af8b315788260205cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:10.506870 16377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00c6acc5246943af8b315788260205cb [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:10.508968 16377 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00c6acc5246943af8b315788260205cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00c6acc5246943af8b315788260205cb" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36783 } }
I20260620 18:43:10.509447 16377 leader_election.cc:302] T 00000000000000000000000000000000 P 00c6acc5246943af8b315788260205cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00c6acc5246943af8b315788260205cb; no voters:
I20260620 18:43:10.510058 16377 leader_election.cc:288] T 00000000000000000000000000000000 P 00c6acc5246943af8b315788260205cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:10.510277 16382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00c6acc5246943af8b315788260205cb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:10.511713 16382 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00c6acc5246943af8b315788260205cb [term 1 LEADER]: Becoming Leader. State: Replica: 00c6acc5246943af8b315788260205cb, State: Running, Role: LEADER
I20260620 18:43:10.512197 16377 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00c6acc5246943af8b315788260205cb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:10.512213 16382 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00c6acc5246943af8b315788260205cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00c6acc5246943af8b315788260205cb" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36783 } }
I20260620 18:43:10.514870 16384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00c6acc5246943af8b315788260205cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00c6acc5246943af8b315788260205cb. Latest consensus state: current_term: 1 leader_uuid: "00c6acc5246943af8b315788260205cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00c6acc5246943af8b315788260205cb" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36783 } } }
I20260620 18:43:10.515242 16383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00c6acc5246943af8b315788260205cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00c6acc5246943af8b315788260205cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00c6acc5246943af8b315788260205cb" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36783 } } }
I20260620 18:43:10.515588 16384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00c6acc5246943af8b315788260205cb [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:10.515611 16383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00c6acc5246943af8b315788260205cb [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:10.517719 16391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:10.523275 16391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:10.532734 16391 catalog_manager.cc:1347] Generated new cluster ID: 6761a15a65724d059dad8d24f61acf0f
I20260620 18:43:10.532866 16391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:10.549865 16391 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:10.551297 16391 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:10.560209 16391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00c6acc5246943af8b315788260205cb: Generated new TSK 0
I20260620 18:43:10.561146 16391 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab.
Entry for principal kudu/127.5.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab.
I20260620 18:43:10.622402 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.129:0
--local_ip_for_outbound_sockets=127.5.225.129
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:36783
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:10.965663 16403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:10.965950 16403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:10.966110 16403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:10.973292 16403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:10.973421 16403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:10.973587 16403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.129
I20260620 18:43:10.983163 16403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:36783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.129
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:43:10.984401 16403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:10.985725 16403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:10.998804 16411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:10.998910 16409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:10.998920 16408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:10.999550 16403 server_base.cc:1034] running on GCE node
I20260620 18:43:11.000110 16403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:11.001024 16403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:11.002257 16403 hybrid_clock.cc:648] HybridClock initialized: now 1781980991002198 us; error 59 us; skew 500 ppm
I20260620 18:43:11.002583 16403 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:43:10 dist-test-slave-k6lg krb5kdc[16297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980990, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:11.006023 16403 init.cc:377] Logged in from keytab as kudu/127.5.225.129@KRBTEST.COM (short username kudu)
I20260620 18:43:11.007457 16403 webserver.cc:460] Webserver started at http://127.5.225.129:39975/ using document root <none> and password file <none>
I20260620 18:43:11.008137 16403 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:11.008270 16403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:11.008580 16403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:11.011569 16403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/data/instance:
uuid: "a3f9e7b33f1049a4899cdad1ee35b73f"
format_stamp: "Formatted at 2026-06-20 18:43:11 on dist-test-slave-k6lg"
server_key: "2ab67d0347f994101e4a6d9400289e10"
server_key_iv: "aba17cc8edc6e7627162aef1416e48c3"
server_key_version: "encryptionkey@0"
I20260620 18:43:11.012359 16403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance:
uuid: "a3f9e7b33f1049a4899cdad1ee35b73f"
format_stamp: "Formatted at 2026-06-20 18:43:11 on dist-test-slave-k6lg"
server_key: "2ab67d0347f994101e4a6d9400289e10"
server_key_iv: "aba17cc8edc6e7627162aef1416e48c3"
server_key_version: "encryptionkey@0"
I20260620 18:43:11.017942 16403 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 18:43:11.021386 16418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:11.022785 16403 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:43:11.022918 16403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/wal
uuid: "a3f9e7b33f1049a4899cdad1ee35b73f"
format_stamp: "Formatted at 2026-06-20 18:43:11 on dist-test-slave-k6lg"
server_key: "2ab67d0347f994101e4a6d9400289e10"
server_key_iv: "aba17cc8edc6e7627162aef1416e48c3"
server_key_version: "encryptionkey@0"
I20260620 18:43:11.023078 16403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:11.044013 16403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:11.048239 16403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:11.048491 16403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:11.049407 16403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:11.050773 16403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:11.050849 16403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:11.050930 16403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:11.050974 16403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:11.085357 16403 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.129:40843
I20260620 18:43:11.085409 16530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.129:40843 every 8 connection(s)
I20260620 18:43:11.086795 16403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
I20260620 18:43:11.087973 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 16403
I20260620 18:43:11.088193 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance
I20260620 18:43:11.089036 6022 external_mini_cluster.cc:1428] Setting key 009c57296dd3be3a346047be2a02b43a
Jun 20 18:43:11 dist-test-slave-k6lg krb5kdc[16297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980990, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:43:11.108204 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:11.089778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:54765 (local address 127.5.225.190:36783)
0620 18:43:11.090292 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:11.090298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:11.091371 (+ 1073us) server_negotiation.cc:407] Connection header received
0620 18:43:11.093135 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:11.093140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:11.093238 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:11.093364 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:11.094926 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:11.095665 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:11.097408 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:11.097805 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:11.097943 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:11.102282 (+ 4339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:11.102288 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:11.102292 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:11.102322 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:11.104980 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:11.105721 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:11.105728 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:11.105737 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:11.105897 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:11.106457 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:11.106462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:11.106473 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:11.106730 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:11.106908 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:11.107600 (+ 692us) server_negotiation.cc:299] Negotiation successful
0620 18:43:11.107833 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":231,"threads_started":1}
I20260620 18:43:11.109184 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:11.090421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36783 (local address 127.5.225.129:54765)
0620 18:43:11.091230 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:11.091268 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:11.092872 (+ 1604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:11.093599 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:11.093608 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:11.094123 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:11.094752 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:11.094769 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:11.095837 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:11.095842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:11.097242 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:11.097258 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:11.098079 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:11.098089 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:11.098464 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:11.099200 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:11.099239 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:11.102012 (+ 2773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:11.105137 (+ 3125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:11.105146 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:11.105163 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:11.105593 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:11.106034 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:11.106038 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:11.106042 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:11.106294 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:11.106880 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:11.106887 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:11.107239 (+ 352us) client_negotiation.cc:769] Sending connection context
0620 18:43:11.107504 (+ 265us) client_negotiation.cc:240] Negotiation successful
0620 18:43:11.108037 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":309,"threads_started":1}
WARNING: no policy specified for kudu/127.5.225.130@KRBTEST.COM; defaulting to no policy
I20260620 18:43:11.110848 16531 heartbeater.cc:347] Connected to a master server at 127.5.225.190:36783
I20260620 18:43:11.111142 16531 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:11.111945 16531 heartbeater.cc:511] Master 127.5.225.190:36783 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.130@KRBTEST.COM" created.
I20260620 18:43:11.114518 16342 ts_manager.cc:194] Registered new tserver with Master: a3f9e7b33f1049a4899cdad1ee35b73f (127.5.225.129:40843)
I20260620 18:43:11.116140 16342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.129@KRBTEST.COM'} at 127.5.225.129:54765
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab.
Entry for principal kudu/127.5.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab.
I20260620 18:43:11.141198 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.130:0
--local_ip_for_outbound_sockets=127.5.225.130
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:36783
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:11.471791 16537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:11.472071 16537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:11.472231 16537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:11.479715 16537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:11.479833 16537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:11.480044 16537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.130
I20260620 18:43:11.489764 16537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:36783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.130
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:43:11.491156 16537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:11.492457 16537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:11.505223 16543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:11.505177 16545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:11.505265 16542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:11.506129 16537 server_base.cc:1034] running on GCE node
I20260620 18:43:11.506713 16537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:11.507642 16537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:11.508870 16537 hybrid_clock.cc:648] HybridClock initialized: now 1781980991508834 us; error 39 us; skew 500 ppm
I20260620 18:43:11.509174 16537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:43:11 dist-test-slave-k6lg krb5kdc[16297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980991, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:11.512291 16537 init.cc:377] Logged in from keytab as kudu/127.5.225.130@KRBTEST.COM (short username kudu)
I20260620 18:43:11.513691 16537 webserver.cc:460] Webserver started at http://127.5.225.130:42297/ using document root <none> and password file <none>
I20260620 18:43:11.514420 16537 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:11.514521 16537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:11.514760 16537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:11.517724 16537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/data/instance:
uuid: "5fe60e555fc348f3a7a4f5f73088b31d"
format_stamp: "Formatted at 2026-06-20 18:43:11 on dist-test-slave-k6lg"
server_key: "0785ab94269e41fd4466dc92cdc7a79b"
server_key_iv: "f976e743aa0e4ae4db39103d62bf897b"
server_key_version: "encryptionkey@0"
I20260620 18:43:11.518520 16537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance:
uuid: "5fe60e555fc348f3a7a4f5f73088b31d"
format_stamp: "Formatted at 2026-06-20 18:43:11 on dist-test-slave-k6lg"
server_key: "0785ab94269e41fd4466dc92cdc7a79b"
server_key_iv: "f976e743aa0e4ae4db39103d62bf897b"
server_key_version: "encryptionkey@0"
I20260620 18:43:11.523633 16537 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 18:43:11.527009 16552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:11.528434 16537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:43:11.528609 16537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/wal
uuid: "5fe60e555fc348f3a7a4f5f73088b31d"
format_stamp: "Formatted at 2026-06-20 18:43:11 on dist-test-slave-k6lg"
server_key: "0785ab94269e41fd4466dc92cdc7a79b"
server_key_iv: "f976e743aa0e4ae4db39103d62bf897b"
server_key_version: "encryptionkey@0"
I20260620 18:43:11.528815 16537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:11.544874 16537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:11.549511 16537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:11.549840 16537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:11.550920 16537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:11.552327 16537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:11.552433 16537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:11.552579 16537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:11.552656 16537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:11.587324 16537 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.130:40387
I20260620 18:43:11.587380 16664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.130:40387 every 8 connection(s)
I20260620 18:43:11.588629 16537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
I20260620 18:43:11.592077 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 16537
I20260620 18:43:11.592279 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance
I20260620 18:43:11.592854 6022 external_mini_cluster.cc:1428] Setting key 2daf81be0cb46bd76e4cf6b8e7ed8db1
Jun 20 18:43:11 dist-test-slave-k6lg krb5kdc[16297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980991, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:43:11.612533 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:11.591398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:58771 (local address 127.5.225.190:36783)
0620 18:43:11.591574 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:11.591579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:11.592988 (+ 1409us) server_negotiation.cc:407] Connection header received
0620 18:43:11.595239 (+ 2251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:11.595243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:11.595306 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:11.595393 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:11.597514 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:11.598380 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:11.600820 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:11.601249 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:11.601355 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:11.605403 (+ 4048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:11.605412 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:11.605420 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:11.605466 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:11.609034 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:11.609842 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:11.609850 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:11.609856 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:11.610008 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:11.610536 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:11.610543 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:11.610549 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:11.610937 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:11.611094 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:11.611807 (+ 713us) server_negotiation.cc:299] Negotiation successful
0620 18:43:11.612072 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:43:11.613605 16667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:11.591944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36783 (local address 127.5.225.130:58771)
0620 18:43:11.592791 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:11.592865 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:11.594917 (+ 2052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:11.595677 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:11.595690 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:11.596488 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:11.597340 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:11.597367 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:11.598532 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:11.598540 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:11.600650 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:11.600667 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:11.601487 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:11.601496 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:11.601823 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:11.602611 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:11.602646 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:11.605167 (+ 2521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:11.609246 (+ 4079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:11.609258 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:11.609281 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:11.609720 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:11.610133 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:11.610139 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:11.610143 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:11.610426 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:11.611070 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:11.611076 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:11.611407 (+ 331us) client_negotiation.cc:769] Sending connection context
0620 18:43:11.611647 (+ 240us) client_negotiation.cc:240] Negotiation successful
0620 18:43:11.612146 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":331,"threads_started":1}
I20260620 18:43:11.615254 16665 heartbeater.cc:347] Connected to a master server at 127.5.225.190:36783
I20260620 18:43:11.615608 16665 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:11.616662 16665 heartbeater.cc:511] Master 127.5.225.190:36783 requested a full tablet report, sending...
I20260620 18:43:11.618494 16342 ts_manager.cc:194] Registered new tserver with Master: 5fe60e555fc348f3a7a4f5f73088b31d (127.5.225.130:40387)
WARNING: no policy specified for kudu/127.5.225.131@KRBTEST.COM; defaulting to no policy
I20260620 18:43:11.619354 16342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.130@KRBTEST.COM'} at 127.5.225.130:58771
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab.
Entry for principal kudu/127.5.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab.
I20260620 18:43:11.649044 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.131:0
--local_ip_for_outbound_sockets=127.5.225.131
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:36783
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:11.970520 16670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:11.970803 16670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:11.970957 16670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:11.978240 16670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:11.978344 16670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:11.978552 16670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.131
I20260620 18:43:11.988229 16670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:36783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.131
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:43:11.989530 16670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:11.990903 16670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:12.004510 16678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:12.004480 16676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:12.004650 16675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:12.005187 16670 server_base.cc:1034] running on GCE node
I20260620 18:43:12.005888 16670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:12.006868 16670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:12.008075 16670 hybrid_clock.cc:648] HybridClock initialized: now 1781980992007999 us; error 66 us; skew 500 ppm
I20260620 18:43:12.008404 16670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:43:12 dist-test-slave-k6lg krb5kdc[16297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980992, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:12.011758 16670 init.cc:377] Logged in from keytab as kudu/127.5.225.131@KRBTEST.COM (short username kudu)
I20260620 18:43:12.013324 16670 webserver.cc:460] Webserver started at http://127.5.225.131:44819/ using document root <none> and password file <none>
I20260620 18:43:12.014099 16670 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:12.014266 16670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:12.014562 16670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:12.017650 16670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/data/instance:
uuid: "06a8c0bbcf5e429392188488d0fa51c8"
format_stamp: "Formatted at 2026-06-20 18:43:12 on dist-test-slave-k6lg"
server_key: "bc44e61a8346ec36a4c524223d5c9e98"
server_key_iv: "ec9fa42dcd9c7332fdb578c5c97eb36d"
server_key_version: "encryptionkey@0"
I20260620 18:43:12.018474 16670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance:
uuid: "06a8c0bbcf5e429392188488d0fa51c8"
format_stamp: "Formatted at 2026-06-20 18:43:12 on dist-test-slave-k6lg"
server_key: "bc44e61a8346ec36a4c524223d5c9e98"
server_key_iv: "ec9fa42dcd9c7332fdb578c5c97eb36d"
server_key_version: "encryptionkey@0"
I20260620 18:43:12.023761 16670 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 18:43:12.027568 16685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:12.029213 16670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 18:43:12.029443 16670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/wal
uuid: "06a8c0bbcf5e429392188488d0fa51c8"
format_stamp: "Formatted at 2026-06-20 18:43:12 on dist-test-slave-k6lg"
server_key: "bc44e61a8346ec36a4c524223d5c9e98"
server_key_iv: "ec9fa42dcd9c7332fdb578c5c97eb36d"
server_key_version: "encryptionkey@0"
I20260620 18:43:12.029704 16670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:12.052496 16670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:12.056852 16670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:12.057168 16670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:12.058233 16670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:12.059684 16670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:12.059759 16670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:12.059930 16670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:12.059995 16670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:12.093205 16670 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.131:40071
I20260620 18:43:12.093255 16797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.131:40071 every 8 connection(s)
I20260620 18:43:12.094450 16670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
I20260620 18:43:12.100906 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 16670
I20260620 18:43:12.101145 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance
I20260620 18:43:12.101583 6022 external_mini_cluster.cc:1428] Setting key 966ecc30a96cc61c8eef0e081776b4b2
Jun 20 18:43:12 dist-test-slave-k6lg krb5kdc[16297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980992, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:43:12.114135 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:12.097080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:51667 (local address 127.5.225.190:36783)
0620 18:43:12.097291 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:12.097297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:12.098530 (+ 1233us) server_negotiation.cc:407] Connection header received
0620 18:43:12.100243 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:12.100248 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:12.100375 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:12.100486 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:12.102086 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:12.103031 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:12.104580 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:12.104976 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:12.105106 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:12.108535 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:12.108540 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:12.108544 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:12.108574 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:12.111063 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:12.111765 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:12.111772 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:12.111775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:12.111907 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:12.112361 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:12.112367 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:12.112370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:12.112681 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:12.112833 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:12.113557 (+ 724us) server_negotiation.cc:299] Negotiation successful
0620 18:43:12.113772 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:43:12.115214 16800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:12.097616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36783 (local address 127.5.225.131:51667)
0620 18:43:12.098373 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:12.098410 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:12.099982 (+ 1572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:12.100712 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:12.100720 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:12.101322 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:12.101927 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:12.101946 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:12.103182 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:12.103188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:12.104438 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:12.104449 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:12.105197 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:12.105205 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:12.105438 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:12.106122 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:12.106144 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:12.108333 (+ 2189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:12.111203 (+ 2870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:12.111211 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:12.111228 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:12.111646 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:12.112016 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:12.112022 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:12.112026 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:12.112250 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:12.112811 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:12.112820 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:12.113155 (+ 335us) client_negotiation.cc:769] Sending connection context
0620 18:43:12.113462 (+ 307us) client_negotiation.cc:240] Negotiation successful
0620 18:43:12.114018 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":266,"threads_started":1}
I20260620 18:43:12.116925 16798 heartbeater.cc:347] Connected to a master server at 127.5.225.190:36783
I20260620 18:43:12.117259 16798 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:12.117921 16798 heartbeater.cc:511] Master 127.5.225.190:36783 requested a full tablet report, sending...
I20260620 18:43:12.119248 16531 heartbeater.cc:503] Master 127.5.225.190:36783 was elected leader, sending a full tablet report...
I20260620 18:43:12.119300 16342 ts_manager.cc:194] Registered new tserver with Master: 06a8c0bbcf5e429392188488d0fa51c8 (127.5.225.131:40071)
I20260620 18:43:12.120252 16342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.131@KRBTEST.COM'} at 127.5.225.131:51667
I20260620 18:43:12.129243 6022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:43:12.145994 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:12.133813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58036 (local address 127.5.225.190:36783)
0620 18:43:12.134008 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:12.134016 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:12.134474 (+ 458us) server_negotiation.cc:407] Connection header received
0620 18:43:12.134670 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:12.134675 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:12.134755 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:12.134847 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:12.136221 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:12.136955 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:12.138288 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:12.138709 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:12.138850 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:12.140999 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:12.141005 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:12.141009 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:12.141049 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:12.143447 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:12.144065 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:12.144071 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:12.144075 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:12.144203 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:12.144565 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:12.144571 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:12.144574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:12.144895 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:12.145156 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:12.145468 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 18:43:12.145687 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 18:43:12.149893 16342 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:43:12.156570 16342 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:43:12.182616 16812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:12.168910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.131:40071 (local address 127.0.0.1:41798)
0620 18:43:12.170716 (+ 1806us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:12.170770 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:12.171005 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:12.172671 (+ 1666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:12.172682 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:12.172722 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:12.173239 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:12.173255 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:12.175892 (+ 2637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:12.175902 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:12.179067 (+ 3165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:12.179087 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:12.181486 (+ 2399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:12.181495 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:12.181693 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:12.181741 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 18:43:12.181903 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 18:43:12.182116 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1429,"thread_start_us":11928,"threads_started":1}
I20260620 18:43:12.183377 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:12.169174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41798 (local address 127.5.225.131:40071)
0620 18:43:12.169969 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:12.169978 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:12.171225 (+ 1247us) server_negotiation.cc:407] Connection header received
0620 18:43:12.171340 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:12.171352 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:12.172093 (+ 741us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:12.172309 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:12.173478 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:12.175683 (+ 2205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:12.179301 (+ 3618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:12.180962 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:12.181088 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:12.181346 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:12.182891 (+ 1545us) server_negotiation.cc:299] Negotiation successful
0620 18:43:12.183066 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":655,"threads_started":1}
I20260620 18:43:12.187565 16733 tablet_service.cc:1467] Processing CreateTablet for tablet d5da4d9cefbf43e2a52855ea2348a393 (DEFAULT_TABLE table=test-table [id=ba1454f393a940e3a4304e119daa465f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:43:12.189057 16733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5da4d9cefbf43e2a52855ea2348a393. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:12.189347 16815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:12.172452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53670 (local address 127.5.225.129:40843)
0620 18:43:12.181698 (+ 9246us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:12.181710 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:12.182502 (+ 792us) server_negotiation.cc:407] Connection header received
0620 18:43:12.182617 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:12.182624 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:12.183118 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:12.183294 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:12.184011 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:12.185139 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:12.187349 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:12.188348 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:12.188498 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:12.188722 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:12.188896 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 18:43:12.189045 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9041,"thread_start_us":8897,"threads_started":1}
I20260620 18:43:12.189392 16814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:12.171867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:40843 (local address 127.0.0.1:53670)
0620 18:43:12.182190 (+ 10323us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:12.182213 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:12.182347 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:12.183402 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:12.183406 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:12.183434 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:12.183832 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:12.183844 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:12.185284 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:12.185290 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:12.187196 (+ 1906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:12.187206 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:12.188547 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:12.188554 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:12.188714 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:12.188754 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 18:43:12.188860 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 18:43:12.188984 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10167,"mutex_wait_us":1,"spinlock_wait_cycles":1920,"thread_start_us":10016,"threads_started":1}
I20260620 18:43:12.191182 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:12.169156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.130:40387 (local address 127.0.0.1:53070)
0620 18:43:12.181130 (+ 11974us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:12.181150 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:12.181361 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:12.182290 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:12.182297 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:12.182344 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:12.182728 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:12.182743 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:12.184572 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:12.184577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:12.186742 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:12.186758 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:12.190341 (+ 3583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:12.190348 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:12.190505 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:12.190536 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 18:43:12.190647 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 18:43:12.190769 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11810,"mutex_wait_us":24,"spinlock_wait_cycles":1792,"thread_start_us":995,"threads_started":1}
I20260620 18:43:12.191926 16813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:12.169687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53070 (local address 127.5.225.130:40387)
0620 18:43:12.180537 (+ 10850us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:12.180547 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:12.181238 (+ 691us) server_negotiation.cc:407] Connection header received
0620 18:43:12.181575 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:12.181585 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:12.181840 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:12.182010 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:12.182927 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:12.184381 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:12.186940 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:12.188346 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:12.188486 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:12.188687 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:12.191428 (+ 2741us) server_negotiation.cc:299] Negotiation successful
0620 18:43:12.191582 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10680,"thread_start_us":10860,"threads_started":1}
I20260620 18:43:12.192627 16466 tablet_service.cc:1467] Processing CreateTablet for tablet d5da4d9cefbf43e2a52855ea2348a393 (DEFAULT_TABLE table=test-table [id=ba1454f393a940e3a4304e119daa465f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:43:12.193694 16466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5da4d9cefbf43e2a52855ea2348a393. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:12.198819 16600 tablet_service.cc:1467] Processing CreateTablet for tablet d5da4d9cefbf43e2a52855ea2348a393 (DEFAULT_TABLE table=test-table [id=ba1454f393a940e3a4304e119daa465f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:43:12.200133 16600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5da4d9cefbf43e2a52855ea2348a393. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:12.203354 16816 tablet_bootstrap.cc:492] T d5da4d9cefbf43e2a52855ea2348a393 P 06a8c0bbcf5e429392188488d0fa51c8: Bootstrap starting.
I20260620 18:43:12.206010 16817 tablet_bootstrap.cc:492] T d5da4d9cefbf43e2a52855ea2348a393 P a3f9e7b33f1049a4899cdad1ee35b73f: Bootstrap starting.
I20260620 18:43:12.207000 16816 tablet_bootstrap.cc:654] T d5da4d9cefbf43e2a52855ea2348a393 P 06a8c0bbcf5e429392188488d0fa51c8: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:12.207998 16816 log.cc:824] T d5da4d9cefbf43e2a52855ea2348a393 P 06a8c0bbcf5e429392188488d0fa51c8: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:12.209381 16817 tablet_bootstrap.cc:654] T d5da4d9cefbf43e2a52855ea2348a393 P a3f9e7b33f1049a4899cdad1ee35b73f: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:12.210660 16817 log.cc:824] T d5da4d9cefbf43e2a52855ea2348a393 P a3f9e7b33f1049a4899cdad1ee35b73f: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:12.212137 16816 tablet_bootstrap.cc:492] T d5da4d9cefbf43e2a52855ea2348a393 P 06a8c0bbcf5e429392188488d0fa51c8: No bootstrap required, opened a new log
I20260620 18:43:12.212430 16816 ts_tablet_manager.cc:1397] T d5da4d9cefbf43e2a52855ea2348a393 P 06a8c0bbcf5e429392188488d0fa51c8: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 18:43:12.213565 16817 tablet_bootstrap.cc:492] T d5da4d9cefbf43e2a52855ea2348a393 P a3f9e7b33f1049a4899cdad1ee35b73f: No bootstrap required, opened a new log
I20260620 18:43:12.213878 16817 ts_tablet_manager.cc:1397] T d5da4d9cefbf43e2a52855ea2348a393 P a3f9e7b33f1049a4899cdad1ee35b73f: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 18:43:12.214059 16820 tablet_bootstrap.cc:492] T d5da4d9cefbf43e2a52855ea2348a393 P 5fe60e555fc348f3a7a4f5f73088b31d: Bootstrap starting.
I20260620 18:43:12.218267 16820 tablet_bootstrap.cc:654] T d5da4d9cefbf43e2a52855ea2348a393 P 5fe60e555fc348f3a7a4f5f73088b31d: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:12.219485 16820 log.cc:824] T d5da4d9cefbf43e2a52855ea2348a393 P 5fe60e555fc348f3a7a4f5f73088b31d: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:12.221760 16817 raft_consensus.cc:348] T d5da4d9cefbf43e2a52855ea2348a393 P a3f9e7b33f1049a4899cdad1ee35b73f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a8c0bbcf5e429392188488d0fa51c8" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 40071 } } peers { permanent_uuid: "5fe60e555fc348f3a7a4f5f73088b31d" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40387 } } peers { permanent_uuid: "a3f9e7b33f1049a4899cdad1ee35b73f" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 40843 } }
I20260620 18:43:12.222121 16817 raft_consensus.cc:374] T d5da4d9cefbf43e2a52855ea2348a393 P a3f9e7b33f1049a4899cdad1ee35b73f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:12.222277 16817 raft_consensus.cc:729] T d5da4d9cefbf43e2a52855ea2348a393 P a3f9e7b33f1049a4899cdad1ee35b73f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3f9e7b33f1049a4899cdad1ee35b73f, State: Initialized, Role: FOLLOWER
I20260620 18:43:12.222404 16820 tablet_bootstrap.cc:492] T d5da4d9cefbf43e2a52855ea2348a393 P 5fe60e555fc348f3a7a4f5f73088b31d: No bootstrap required, opened a new log
I20260620 18:43:12.222668 16820 ts_tablet_manager.cc:1397] T d5da4d9cefbf43e2a52855ea2348a393 P 5fe60e555fc348f3a7a4f5f73088b31d: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 18:43:12.222896 16817 consensus_queue.cc:260] T d5da4d9cefbf43e2a52855ea2348a393 P a3f9e7b33f1049a4899cdad1ee35b73f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a8c0bbcf5e429392188488d0fa51c8" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 40071 } } peers { permanent_uuid: "5fe60e555fc348f3a7a4f5f73088b31d" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40387 } } peers { permanent_uuid: "a3f9e7b33f1049a4899cdad1ee35b73f" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 40843 } }
I20260620 18:43:12.222785 16816 raft_consensus.cc:348] T d5da4d9cefbf43e2a52855ea2348a393 P 06a8c0bbcf5e429392188488d0fa51c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a8c0bbcf5e429392188488d0fa51c8" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 40071 } } peers { permanent_uuid: "5fe60e555fc348f3a7a4f5f73088b31d" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40387 } } peers { permanent_uuid: "a3f9e7b33f1049a4899cdad1ee35b73f" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 40843 } }
I20260620 18:43:12.223239 16816 raft_consensus.cc:374] T d5da4d9cefbf43e2a52855ea2348a393 P 06a8c0bbcf5e429392188488d0fa51c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:12.223382 16816 raft_consensus.cc:729] T d5da4d9cefbf43e2a52855ea2348a393 P 06a8c0bbcf5e429392188488d0fa51c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06a8c0bbcf5e429392188488d0fa51c8, State: Initialized, Role: FOLLOWER
I20260620 18:43:12.224392 16817 ts_tablet_manager.cc:1428] T d5da4d9cefbf43e2a52855ea2348a393 P a3f9e7b33f1049a4899cdad1ee35b73f: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
I20260620 18:43:12.224891 16816 consensus_queue.cc:260] T d5da4d9cefbf43e2a52855ea2348a393 P 06a8c0bbcf5e429392188488d0fa51c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a8c0bbcf5e429392188488d0fa51c8" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 40071 } } peers { permanent_uuid: "5fe60e555fc348f3a7a4f5f73088b31d" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40387 } } peers { permanent_uuid: "a3f9e7b33f1049a4899cdad1ee35b73f" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 40843 } }
I20260620 18:43:12.226493 16816 ts_tablet_manager.cc:1428] T d5da4d9cefbf43e2a52855ea2348a393 P 06a8c0bbcf5e429392188488d0fa51c8: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260620 18:43:12.226881 16798 heartbeater.cc:503] Master 127.5.225.190:36783 was elected leader, sending a full tablet report...
I20260620 18:43:12.231967 16820 raft_consensus.cc:348] T d5da4d9cefbf43e2a52855ea2348a393 P 5fe60e555fc348f3a7a4f5f73088b31d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a8c0bbcf5e429392188488d0fa51c8" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 40071 } } peers { permanent_uuid: "5fe60e555fc348f3a7a4f5f73088b31d" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40387 } } peers { permanent_uuid: "a3f9e7b33f1049a4899cdad1ee35b73f" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 40843 } }
I20260620 18:43:12.232327 16820 raft_consensus.cc:374] T d5da4d9cefbf43e2a52855ea2348a393 P 5fe60e555fc348f3a7a4f5f73088b31d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:12.232445 16820 raft_consensus.cc:729] T d5da4d9cefbf43e2a52855ea2348a393 P 5fe60e555fc348f3a7a4f5f73088b31d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fe60e555fc348f3a7a4f5f73088b31d, State: Initialized, Role: FOLLOWER
I20260620 18:43:12.233017 16820 consensus_queue.cc:260] T d5da4d9cefbf43e2a52855ea2348a393 P 5fe60e555fc348f3a7a4f5f73088b31d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a8c0bbcf5e429392188488d0fa51c8" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 40071 } } peers { permanent_uuid: "5fe60e555fc348f3a7a4f5f73088b31d" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40387 } } peers { permanent_uuid: "a3f9e7b33f1049a4899cdad1ee35b73f" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 40843 } }
I20260620 18:43:12.234078 16665 heartbeater.cc:503] Master 127.5.225.190:36783 was elected leader, sending a full tablet report...
I20260620 18:43:12.234583 16820 ts_tablet_manager.cc:1428] T d5da4d9cefbf43e2a52855ea2348a393 P 5fe60e555fc348f3a7a4f5f73088b31d: Time spent starting tablet: real 0.012s user 0.005s sys 0.007s
W20260620 18:43:12.339439 16532 tablet.cc:2369] T d5da4d9cefbf43e2a52855ea2348a393 P a3f9e7b33f1049a4899cdad1ee35b73f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:43:12.340566 16666 tablet.cc:2369] T d5da4d9cefbf43e2a52855ea2348a393 P 5fe60e555fc348f3a7a4f5f73088b31d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:43:12.343657 16824 raft_consensus.cc:482] T d5da4d9cefbf43e2a52855ea2348a393 P 5fe60e555fc348f3a7a4f5f73088b31d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:43:12.343940 16824 raft_consensus.cc:504] T d5da4d9cefbf43e2a52855ea2348a393 P 5fe60e555fc348f3a7a4f5f73088b31d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a8c0bbcf5e429392188488d0fa51c8" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 40071 } } peers { permanent_uuid: "5fe60e555fc348f3a7a4f5f73088b31d" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40387 } } peers { permanent_uuid: "a3f9e7b33f1049a4899cdad1ee35b73f" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 40843 } }
I20260620 18:43:12.345643 16824 leader_election.cc:288] T d5da4d9cefbf43e2a52855ea2348a393 P 5fe60e555fc348f3a7a4f5f73088b31d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06a8c0bbcf5e429392188488d0fa51c8 (127.5.225.131:40071), a3f9e7b33f1049a4899cdad1ee35b73f (127.5.225.129:40843)
W20260620 18:43:12.346350 16799 tablet.cc:2369] T d5da4d9cefbf43e2a52855ea2348a393 P 06a8c0bbcf5e429392188488d0fa51c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:43:12.355090 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:12.346108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:54723 (local address 127.5.225.131:40071)
0620 18:43:12.346358 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:12.346366 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:12.346964 (+ 598us) server_negotiation.cc:407] Connection header received
0620 18:43:12.347339 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:12.347343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:12.347437 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:12.347536 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:12.348195 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:12.349358 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:12.352505 (+ 3147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:12.353750 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:12.353917 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:12.353992 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:12.354701 (+ 709us) server_negotiation.cc:299] Negotiation successful
0620 18:43:12.354834 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 18:43:12.355101 16815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:12.346746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:38323 (local address 127.5.225.129:40843)
0620 18:43:12.346950 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:12.346956 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:12.347408 (+ 452us) server_negotiation.cc:407] Connection header received
0620 18:43:12.347537 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:12.347542 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:12.347613 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:12.347716 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:12.348377 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:12.349423 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:12.352499 (+ 3076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:12.353768 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:12.353917 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:12.353992 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:12.354701 (+ 709us) server_negotiation.cc:299] Negotiation successful
0620 18:43:12.354853 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:43:12.355322 16826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:12.346268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:40843 (local address 127.5.225.130:38323)
0620 18:43:12.347306 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:12.347320 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:12.347423 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:12.347742 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:12.347746 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:12.347774 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:12.348236 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:12.348244 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:12.349543 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:12.349550 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:12.352326 (+ 2776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:12.352341 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:12.354030 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:12.354037 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:12.354409 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:12.354561 (+ 152us) client_negotiation.cc:769] Sending connection context
0620 18:43:12.354658 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 18:43:12.354817 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":937,"mutex_wait_us":389,"thread_start_us":713,"threads_started":1}
I20260620 18:43:12.355345 16825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:12.346060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.131:40071 (local address 127.5.225.130:54723)
0620 18:43:12.346851 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:12.346870 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:12.347105 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:12.347576 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:12.347579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:12.347597 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:12.348023 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:12.348041 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:12.349512 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:12.349519 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:12.352329 (+ 2810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:12.352341 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:12.354034 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:12.354039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:12.354410 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:12.354561 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 18:43:12.354658 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 18:43:12.354817 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"mutex_wait_us":328,"thread_start_us":468,"threads_started":1}
I20260620 18:43:12.356467 16753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5da4d9cefbf43e2a52855ea2348a393" candidate_uuid: "5fe60e555fc348f3a7a4f5f73088b31d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06a8c0bbcf5e429392188488d0fa51c8" is_pre_election: true
I20260620 18:43:12.356483 16486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5da4d9cefbf43e2a52855ea2348a393" candidate_uuid: "5fe60e555fc348f3a7a4f5f73088b31d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3f9e7b33f1049a4899cdad1ee35b73f" is_pre_election: true
I20260620 18:43:12.356978 16753 raft_consensus.cc:2454] T d5da4d9cefbf43e2a52855ea2348a393 P 06a8c0bbcf5e429392188488d0fa51c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fe60e555fc348f3a7a4f5f73088b31d in term 0.
I20260620 18:43:12.356978 16486 raft_consensus.cc:2454] T d5da4d9cefbf43e2a52855ea2348a393 P a3f9e7b33f1049a4899cdad1ee35b73f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fe60e555fc348f3a7a4f5f73088b31d in term 0.
I20260620 18:43:12.357632 16556 leader_election.cc:302] T d5da4d9cefbf43e2a52855ea2348a393 P 5fe60e555fc348f3a7a4f5f73088b31d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fe60e555fc348f3a7a4f5f73088b31d, a3f9e7b33f1049a4899cdad1ee35b73f; no voters:
I20260620 18:43:12.358029 16824 raft_consensus.cc:2781] T d5da4d9cefbf43e2a52855ea2348a393 P 5fe60e555fc348f3a7a4f5f73088b31d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:43:12.358170 16824 raft_consensus.cc:482] T d5da4d9cefbf43e2a52855ea2348a393 P 5fe60e555fc348f3a7a4f5f73088b31d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:43:12.358319 16824 raft_consensus.cc:3037] T d5da4d9cefbf43e2a52855ea2348a393 P 5fe60e555fc348f3a7a4f5f73088b31d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:12.360450 16824 raft_consensus.cc:504] T d5da4d9cefbf43e2a52855ea2348a393 P 5fe60e555fc348f3a7a4f5f73088b31d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a8c0bbcf5e429392188488d0fa51c8" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 40071 } } peers { permanent_uuid: "5fe60e555fc348f3a7a4f5f73088b31d" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40387 } } peers { permanent_uuid: "a3f9e7b33f1049a4899cdad1ee35b73f" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 40843 } }
I20260620 18:43:12.361079 16824 leader_election.cc:288] T d5da4d9cefbf43e2a52855ea2348a393 P 5fe60e555fc348f3a7a4f5f73088b31d [CANDIDATE]: Term 1 election: Requested vote from peers 06a8c0bbcf5e429392188488d0fa51c8 (127.5.225.131:40071), a3f9e7b33f1049a4899cdad1ee35b73f (127.5.225.129:40843)
I20260620 18:43:12.361490 16753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5da4d9cefbf43e2a52855ea2348a393" candidate_uuid: "5fe60e555fc348f3a7a4f5f73088b31d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06a8c0bbcf5e429392188488d0fa51c8"
I20260620 18:43:12.361583 16486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5da4d9cefbf43e2a52855ea2348a393" candidate_uuid: "5fe60e555fc348f3a7a4f5f73088b31d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3f9e7b33f1049a4899cdad1ee35b73f"
I20260620 18:43:12.361706 16753 raft_consensus.cc:3037] T d5da4d9cefbf43e2a52855ea2348a393 P 06a8c0bbcf5e429392188488d0fa51c8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:12.361761 16486 raft_consensus.cc:3037] T d5da4d9cefbf43e2a52855ea2348a393 P a3f9e7b33f1049a4899cdad1ee35b73f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:12.364926 16486 raft_consensus.cc:2454] T d5da4d9cefbf43e2a52855ea2348a393 P a3f9e7b33f1049a4899cdad1ee35b73f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fe60e555fc348f3a7a4f5f73088b31d in term 1.
I20260620 18:43:12.364926 16753 raft_consensus.cc:2454] T d5da4d9cefbf43e2a52855ea2348a393 P 06a8c0bbcf5e429392188488d0fa51c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fe60e555fc348f3a7a4f5f73088b31d in term 1.
I20260620 18:43:12.365346 16556 leader_election.cc:302] T d5da4d9cefbf43e2a52855ea2348a393 P 5fe60e555fc348f3a7a4f5f73088b31d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 06a8c0bbcf5e429392188488d0fa51c8, 5fe60e555fc348f3a7a4f5f73088b31d, a3f9e7b33f1049a4899cdad1ee35b73f; no voters:
I20260620 18:43:12.365734 16824 raft_consensus.cc:2781] T d5da4d9cefbf43e2a52855ea2348a393 P 5fe60e555fc348f3a7a4f5f73088b31d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:12.366261 16824 raft_consensus.cc:686] T d5da4d9cefbf43e2a52855ea2348a393 P 5fe60e555fc348f3a7a4f5f73088b31d [term 1 LEADER]: Becoming Leader. State: Replica: 5fe60e555fc348f3a7a4f5f73088b31d, State: Running, Role: LEADER
I20260620 18:43:12.366768 16824 consensus_queue.cc:237] T d5da4d9cefbf43e2a52855ea2348a393 P 5fe60e555fc348f3a7a4f5f73088b31d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a8c0bbcf5e429392188488d0fa51c8" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 40071 } } peers { permanent_uuid: "5fe60e555fc348f3a7a4f5f73088b31d" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40387 } } peers { permanent_uuid: "a3f9e7b33f1049a4899cdad1ee35b73f" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 40843 } }
I20260620 18:43:12.371671 16340 catalog_manager.cc:5495] T d5da4d9cefbf43e2a52855ea2348a393 P 5fe60e555fc348f3a7a4f5f73088b31d reported cstate change: term changed from 0 to 1, leader changed from <none> to 5fe60e555fc348f3a7a4f5f73088b31d (127.5.225.130). New cstate: current_term: 1 leader_uuid: "5fe60e555fc348f3a7a4f5f73088b31d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a8c0bbcf5e429392188488d0fa51c8" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 40071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fe60e555fc348f3a7a4f5f73088b31d" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3f9e7b33f1049a4899cdad1ee35b73f" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 40843 } health_report { overall_health: UNKNOWN } } }
I20260620 18:43:12.429884 16813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:12.423429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53078 (local address 127.5.225.130:40387)
0620 18:43:12.423704 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:12.423710 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:12.423732 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:43:12.423855 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:12.423859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:12.423988 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:12.424163 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:43:12.424663 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:12.425801 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:12.427469 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:12.428003 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:12.428157 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:12.428647 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:43:12.429170 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:43:12.429345 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:12.429398 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 18:43:12.429480 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 18:43:12.439016 16753 raft_consensus.cc:1261] T d5da4d9cefbf43e2a52855ea2348a393 P 06a8c0bbcf5e429392188488d0fa51c8 [term 1 FOLLOWER]: Refusing update from remote peer 5fe60e555fc348f3a7a4f5f73088b31d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:43:12.439016 16486 raft_consensus.cc:1261] T d5da4d9cefbf43e2a52855ea2348a393 P a3f9e7b33f1049a4899cdad1ee35b73f [term 1 FOLLOWER]: Refusing update from remote peer 5fe60e555fc348f3a7a4f5f73088b31d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:43:12.440142 16827 consensus_queue.cc:1035] T d5da4d9cefbf43e2a52855ea2348a393 P 5fe60e555fc348f3a7a4f5f73088b31d [LEADER]: Connected to new peer: Peer: permanent_uuid: "06a8c0bbcf5e429392188488d0fa51c8" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 40071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:43:12.440538 16824 consensus_queue.cc:1035] T d5da4d9cefbf43e2a52855ea2348a393 P 5fe60e555fc348f3a7a4f5f73088b31d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3f9e7b33f1049a4899cdad1ee35b73f" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 40843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:43:12.467017 16342 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:43:12.467342 16342 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:43:12.471127 16342 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 00c6acc5246943af8b315788260205cb: Sending DeleteTablet for 3 replicas of tablet d5da4d9cefbf43e2a52855ea2348a393
I20260620 18:43:12.472234 16733 tablet_service.cc:1514] Processing DeleteTablet for tablet d5da4d9cefbf43e2a52855ea2348a393 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:43:12 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:41798
I20260620 18:43:12.473316 16839 tablet_replica.cc:330] T d5da4d9cefbf43e2a52855ea2348a393 P 06a8c0bbcf5e429392188488d0fa51c8: stopping tablet replica
I20260620 18:43:12.473804 16839 raft_consensus.cc:2229] T d5da4d9cefbf43e2a52855ea2348a393 P 06a8c0bbcf5e429392188488d0fa51c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:43:12.474344 16839 raft_consensus.cc:2258] T d5da4d9cefbf43e2a52855ea2348a393 P 06a8c0bbcf5e429392188488d0fa51c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:43:12.474421 16600 tablet_service.cc:1514] Processing DeleteTablet for tablet d5da4d9cefbf43e2a52855ea2348a393 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:43:12 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:53070
I20260620 18:43:12.474735 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 16403
I20260620 18:43:12.474682 16466 tablet_service.cc:1514] Processing DeleteTablet for tablet d5da4d9cefbf43e2a52855ea2348a393 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:43:12 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:53670
I20260620 18:43:12.477846 16839 ts_tablet_manager.cc:1905] T d5da4d9cefbf43e2a52855ea2348a393 P 06a8c0bbcf5e429392188488d0fa51c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:43:12.485787 16839 ts_tablet_manager.cc:1918] T d5da4d9cefbf43e2a52855ea2348a393 P 06a8c0bbcf5e429392188488d0fa51c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:43:12.486128 16839 log.cc:1196] T d5da4d9cefbf43e2a52855ea2348a393 P 06a8c0bbcf5e429392188488d0fa51c8: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-2/wal/wals/d5da4d9cefbf43e2a52855ea2348a393
I20260620 18:43:12.486829 16839 ts_tablet_manager.cc:1939] T d5da4d9cefbf43e2a52855ea2348a393 P 06a8c0bbcf5e429392188488d0fa51c8: Deleting consensus metadata
I20260620 18:43:12.487053 16841 tablet_replica.cc:330] T d5da4d9cefbf43e2a52855ea2348a393 P a3f9e7b33f1049a4899cdad1ee35b73f: stopping tablet replica
I20260620 18:43:12.487552 16841 raft_consensus.cc:2229] T d5da4d9cefbf43e2a52855ea2348a393 P a3f9e7b33f1049a4899cdad1ee35b73f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:43:12.488097 16841 raft_consensus.cc:2258] T d5da4d9cefbf43e2a52855ea2348a393 P a3f9e7b33f1049a4899cdad1ee35b73f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:43:12.489140 16327 catalog_manager.cc:4841] TS 06a8c0bbcf5e429392188488d0fa51c8 (127.5.225.131:40071): tablet d5da4d9cefbf43e2a52855ea2348a393 (table test-table [id=ba1454f393a940e3a4304e119daa465f]) successfully deleted
I20260620 18:43:12.490513 16841 ts_tablet_manager.cc:1905] T d5da4d9cefbf43e2a52855ea2348a393 P a3f9e7b33f1049a4899cdad1ee35b73f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:43:12.491401 16840 tablet_replica.cc:330] T d5da4d9cefbf43e2a52855ea2348a393 P 5fe60e555fc348f3a7a4f5f73088b31d: stopping tablet replica
I20260620 18:43:12.492013 16840 raft_consensus.cc:2229] T d5da4d9cefbf43e2a52855ea2348a393 P 5fe60e555fc348f3a7a4f5f73088b31d [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:43:12.492825 16840 raft_consensus.cc:2258] T d5da4d9cefbf43e2a52855ea2348a393 P 5fe60e555fc348f3a7a4f5f73088b31d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:43:12.495798 16840 ts_tablet_manager.cc:1905] T d5da4d9cefbf43e2a52855ea2348a393 P 5fe60e555fc348f3a7a4f5f73088b31d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:43:12.497605 16841 ts_tablet_manager.cc:1918] T d5da4d9cefbf43e2a52855ea2348a393 P a3f9e7b33f1049a4899cdad1ee35b73f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:43:12.498011 16841 log.cc:1196] T d5da4d9cefbf43e2a52855ea2348a393 P a3f9e7b33f1049a4899cdad1ee35b73f: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-0/wal/wals/d5da4d9cefbf43e2a52855ea2348a393
I20260620 18:43:12.498628 16841 ts_tablet_manager.cc:1939] T d5da4d9cefbf43e2a52855ea2348a393 P a3f9e7b33f1049a4899cdad1ee35b73f: Deleting consensus metadata
I20260620 18:43:12.500519 16329 catalog_manager.cc:4841] TS a3f9e7b33f1049a4899cdad1ee35b73f (127.5.225.129:40843): tablet d5da4d9cefbf43e2a52855ea2348a393 (table test-table [id=ba1454f393a940e3a4304e119daa465f]) successfully deleted
Jun 20 18:43:12 dist-test-slave-k6lg krb5kdc[16297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.129@KRBTEST.COM
I20260620 18:43:12.505511 16840 ts_tablet_manager.cc:1918] T d5da4d9cefbf43e2a52855ea2348a393 P 5fe60e555fc348f3a7a4f5f73088b31d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:43:12.505713 16840 log.cc:1196] T d5da4d9cefbf43e2a52855ea2348a393 P 5fe60e555fc348f3a7a4f5f73088b31d: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980894474392-6022-0/minicluster-data/ts-1/wal/wals/d5da4d9cefbf43e2a52855ea2348a393
I20260620 18:43:12.506292 16840 ts_tablet_manager.cc:1939] T d5da4d9cefbf43e2a52855ea2348a393 P 5fe60e555fc348f3a7a4f5f73088b31d: Deleting consensus metadata
I20260620 18:43:12.508073 16328 catalog_manager.cc:4841] TS 5fe60e555fc348f3a7a4f5f73088b31d (127.5.225.130:40387): tablet d5da4d9cefbf43e2a52855ea2348a393 (table test-table [id=ba1454f393a940e3a4304e119daa465f]) successfully deleted
I20260620 18:43:12.510816 16815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:12.486200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53672 (local address 127.5.225.129:40843)
0620 18:43:12.486570 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:12.486578 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:12.490684 (+ 4106us) server_negotiation.cc:407] Connection header received
0620 18:43:12.490948 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:12.490955 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:12.491047 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:12.491156 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:12.493084 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:12.493863 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:12.498303 (+ 4440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:12.498993 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:12.499103 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:12.503983 (+ 4880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:12.503996 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:12.504015 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:12.504056 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:12.507819 (+ 3763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:12.508697 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:12.508706 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:12.508710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:12.508841 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:12.509227 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:12.509232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:12.509235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:12.509569 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:12.509893 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:12.510222 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 18:43:12.510438 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210}
I20260620 18:43:12.511510 16526 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:12.647370 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 16403
I20260620 18:43:12.664295 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 16537
Jun 20 18:43:12 dist-test-slave-k6lg krb5kdc[16297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.130@KRBTEST.COM
I20260620 18:43:12.680397 16813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:12.665083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53094 (local address 127.5.225.130:40387)
0620 18:43:12.665267 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:12.665274 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:12.665299 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:43:12.665524 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:12.665529 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:12.665614 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:12.665734 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:12.667008 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:12.667832 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:12.669179 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:12.669642 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:12.669786 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:12.674028 (+ 4242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:12.674041 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:12.674056 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:12.674100 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:12.677490 (+ 3390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:12.678294 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:12.678301 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:12.678304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:12.678442 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:12.678811 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:12.678815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:12.678817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:12.679205 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:12.679441 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:12.679743 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 18:43:12.679950 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 18:43:12.681167 16660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:12.796777 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 16537
I20260620 18:43:12.815591 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 16670
Jun 20 18:43:12 dist-test-slave-k6lg krb5kdc[16297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.131@KRBTEST.COM
I20260620 18:43:12.830302 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:12.816406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41812 (local address 127.5.225.131:40071)
0620 18:43:12.816576 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:12.816581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:12.816660 (+ 79us) server_negotiation.cc:407] Connection header received
0620 18:43:12.816804 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:12.816808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:12.816870 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:12.816955 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:12.818394 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:12.819092 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:12.820236 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:12.820599 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:12.820691 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:12.825018 (+ 4327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:12.825032 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:12.825048 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:12.825086 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:12.827794 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:12.828428 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:12.828436 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:12.828438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:12.828586 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:12.828945 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:12.828949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:12.828951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:12.829276 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:12.829496 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:12.829717 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 18:43:12.829879 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:43:12.830927 16793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:12.917516 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 16670
I20260620 18:43:12.933941 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 16310
I20260620 18:43:12.935256 16372 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:13.043404 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 16310
2026-06-20T18:43:13Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4329 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:13 dist-test-slave-k6lg krb5kdc[16849](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:13 dist-test-slave-k6lg krb5kdc[16849](info): set up 2 sockets
Jun 20 18:43:13 dist-test-slave-k6lg krb5kdc[16849](info): commencing operation
krb5kdc: starting...
W20260620 18:43:14.179554 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.006s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:43:14 dist-test-slave-k6lg krb5kdc[16849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:14Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.5.225.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.5.225.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.5.225.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/krb5kdc/rangeradmin_127.5.225.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.5.225.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/krb5kdc/rangeradmin_127.5.225.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.5.225.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.5.225.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.5.225.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/krb5kdc/rangerlookup_127.5.225.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.5.225.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/krb5kdc/rangerlookup_127.5.225.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.225.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.225.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.225.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/krb5kdc/HTTP_127.5.225.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.225.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/krb5kdc/HTTP_127.5.225.148@KRBTEST.COM.keytab.
I20260620 18:43:14.508921 6022 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskuQUw40/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/postgres -l logfile start
2026-06-20 18:43:17.002 UTC [16879] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 18:43:17.002 UTC [16879] LOG: listening on IPv4 address "127.5.225.148", port 54953
2026-06-20 18:43:17.006 UTC [16879] LOG: listening on Unix socket "/tmp/.s.PGSQL.54953"
2026-06-20 18:43:17.029 UTC [16882] LOG: database system was shut down at 2026-06-20 18:43:15 UTC
2026-06-20 18:43:17.033 UTC [16879] LOG: database system is ready to accept connections
I20260620 18:43:18.006699 6022 mini_postgres.cc:96] Postgres bound to 54953
2026-06-20 18:43:18.024 UTC [16890] FATAL: database "slave" does not exist
127.5.225.148:54953 - accepting connections
I20260620 18:43:18.025628 6022 mini_ranger.cc:161] Starting Ranger...
I20260620 18:43:18.051664 6022 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 18:43:18.540972 6022 mini_ranger.cc:87] Created ranger Postgres database
I20260620 18:43:18.541107 6022 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ranger-admin
2026-06-20 18:43:18,845 [I] DB FLAVOR :POSTGRES
2026-06-20 18:43:18,845 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 18:43:18,845 [I] Checking connection..
2026-06-20 18:43:18,845 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:54953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 18:43:19,109 [I] Checking connection passed.
2026-06-20 18:43:19,109 [I] --------- Verifying version history table ---------
2026-06-20 18:43:19,110 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:54953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:43:19,349 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 18:43:19,349 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:54953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:43:19,624 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 18:43:19,624 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 18:43:19,624 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:54953/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 18:43:19.854 UTC [16954] WARNING: there is no transaction in progress
2026-06-20 18:43:19,860 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 18:43:19,860 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:54953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:43:20,090 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 18:43:20,091 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 18:43:20,091 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:54953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 18:43:20,346 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:54953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 18:43:20,583 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:54953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-k6lg.c.gcp-upstream.internal','N') ;"
2026-06-20 18:43:20,836 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:54953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 18:43:21,089 [I] Table x_portal_user does not exist in database ranger
2026-06-20 18:43:21,089 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:54953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 18:43:21,330 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 18:43:21,330 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:54953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 18:43:21,583 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:54953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 18:43:21,820 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 18:43:21,820 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:54953/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 18:43:22.775 UTC [17062] WARNING: there is no transaction in progress
2026-06-20 18:43:22.791 UTC [17062] WARNING: there is no transaction in progress
2026-06-20 18:43:22.807 UTC [17062] WARNING: there is no transaction in progress
2026-06-20 18:43:22.822 UTC [17062] WARNING: there is no transaction in progress
2026-06-20 18:43:22.838 UTC [17062] WARNING: there is no transaction in progress
2026-06-20 18:43:22.858 UTC [17062] WARNING: there is no transaction in progress
2026-06-20 18:43:22.911 UTC [17062] WARNING: there is no transaction in progress
2026-06-20 18:43:22.917 UTC [17062] WARNING: there is no transaction in progress
2026-06-20 18:43:22.932 UTC [17062] WARNING: there is no transaction in progress
2026-06-20 18:43:22.942 UTC [17062] WARNING: there is no transaction in progress
2026-06-20 18:43:22.952 UTC [17062] WARNING: there is no transaction in progress
2026-06-20 18:43:22.965 UTC [17062] WARNING: there is no transaction in progress
2026-06-20 18:43:22.974 UTC [17062] WARNING: there is no transaction in progress
2026-06-20 18:43:22.981 UTC [17062] WARNING: there is no transaction in progress
2026-06-20 18:43:22.988 UTC [17062] WARNING: there is no transaction in progress
2026-06-20 18:43:22.995 UTC [17062] WARNING: there is no transaction in progress
2026-06-20 18:43:23,630 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 18:43:23,630 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:54953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 18:43:23,849 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 18:43:23,849 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:54953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 18:43:24,111 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 18:43:24,111 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:54953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 18:43:24,363 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 18:43:24,363 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:54953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 18:43:24,623 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:54953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 18:43:24,872 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:54953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-k6lg.c.gcp-upstream.internal';"
2026-06-20 18:43:25,107 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 18:43:25,108 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:54953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 18:43:25,330 [I] DB_PATCHES have already been applied
I20260620 18:43:25.336406 6022 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ranger-admin:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskuQUw40/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 18:43:25.336519 6022 mini_ranger.cc:193] Using host: 127.5.225.148
I20260620 18:43:25.350905 6022 mini_ranger.cc:227] Ranger admin URL: http://127.5.225.148:35695
Jun 20, 2026 6:43:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:43:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:43:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 6:43:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 6:43:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 6:43:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 6:43:26 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 18:43:26.489711 6022 mini_ranger.cc:160] Time spent starting Ranger: real 8.464s user 0.000s sys 0.039s
Jun 20, 2026 6:43:45 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 6:43:45 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 6:43:45 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 6:43:45 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 6:43:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 6:43:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 6:43:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 6:43:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 6:43:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 6:43:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 6:43:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 6:43:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 6:43:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 6:43:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 6:43:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 6:43:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 6:43:45 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 6:43:46 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 18:43:47.040073 6022 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.5.225.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.5.225.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.5.225.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/krb5kdc/rangerkms_127.5.225.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.5.225.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/krb5kdc/rangerkms_127.5.225.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.225.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.5.225.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.225.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/krb5kdc/HTTP_127.5.225.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.225.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/krb5kdc/HTTP_127.5.225.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 18:43:47 dist-test-slave-k6lg krb5kdc[16849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981027, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 18:43:47.205592 6022 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 18:43:47.226773 6022 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 18:43:48.084162 6022 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 18:43:48.084345 6022 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ranger-kms
I20260620 18:43:48.084493 6022 mini_ranger_kms.cc:227] Using postgres at 127.5.225.148:54953
2026-06-20 18:43:48,497 [I] DB FLAVOR :POSTGRES
2026-06-20 18:43:48,498 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 18:43:48,498 [I] Checking connection
2026-06-20 18:43:48,498 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:54953/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 18:43:48,748 [I] connection success
2026-06-20 18:43:48,748 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 18:43:48,748 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 18:43:48,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:54953/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 18:43:48,983 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 18:43:48,983 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 18:43:48,983 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 18:43:48,983 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:54953/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 18:43:49,236 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 18:43:49.675905 6022 mini_ranger_kms.cc:310] Created kms service
I20260620 18:43:49.868072 6022 mini_ranger_kms.cc:326] Created kudu user
I20260620 18:43:49.912613 6022 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 18:43:49 dist-test-slave-k6lg krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981027, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.5.225.148@KRBTEST.COM
I20260620 18:43:50.320896 6022 mini_ranger_kms.cc:384] Added ranger policy
I20260620 18:43:50.321214 6022 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ranger-kms:/tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 18:43:50.321265 6022 mini_ranger_kms.cc:240] Using host: 127.5.225.148
I20260620 18:43:50.341244 6022 mini_ranger_kms.cc:276] Ranger KMS PID: 17242
I20260620 18:43:50.341436 6022 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.5.225.148:33185
Jun 20, 2026 6:43:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:43:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:43:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 6:43:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 6:43:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 6:43:51 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 18:43:52.650449 6022 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.445s user 0.157s sys 0.279s
I20260620 18:43:52.650679 6022 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 18:43:52.650885 6022 mini_ranger_kms.cc:401] 127.5.225.148:33185/kms/v1/keys
Jun 20 18:43:52 dist-test-slave-k6lg krb5kdc[16849](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781981032, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.225.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 18:43:53 dist-test-slave-k6lg krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981032, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.225.148@KRBTEST.COM for HTTP/127.5.225.148@KRBTEST.COM
Jun 20 18:43:53 dist-test-slave-k6lg krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981032, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.225.148@KRBTEST.COM for HTTP/127.5.225.148@KRBTEST.COM
Jun 20 18:43:55 dist-test-slave-k6lg krb5kdc[16849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.5.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
I20260620 18:43:55.881587 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:37695
--webserver_interface=127.5.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:36279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:37695
--ranger_config_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.225.148:33185/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:56.212582 17293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:56.212955 17293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:56.213126 17293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:56.220202 17293 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:56.220358 17293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:56.220469 17293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:56.220572 17293 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:56.220662 17293 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:56.230633 17293 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:36279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/master-0/wal
--ranger_kms_url=127.5.225.148:33185/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:37695
--ranger_config_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:37695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:43:56.231990 17293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:56.233294 17293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:56.245255 17298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:56.245281 17299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:56.245210 17301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:56.246053 17293 server_base.cc:1034] running on GCE node
I20260620 18:43:56.246843 17293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:56.248205 17293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:56.249460 17293 hybrid_clock.cc:648] HybridClock initialized: now 1781981036249420 us; error 38 us; skew 500 ppm
I20260620 18:43:56.249811 17293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:43:56 dist-test-slave-k6lg krb5kdc[16849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981036, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:56.252942 17293 init.cc:377] Logged in from keytab as kudu/127.5.225.190@KRBTEST.COM (short username kudu)
I20260620 18:43:56.254431 17293 webserver.cc:460] Webserver started at http://127.5.225.190:46323/ using document root <none> and password file <none>
I20260620 18:43:56.255043 17293 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:56.255175 17293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:56.255470 17293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:43:56 dist-test-slave-k6lg krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981036, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for HTTP/127.5.225.148@KRBTEST.COM
I20260620 18:43:56.306438 17293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "44c41159cbe44ea58b3f885159cb95e5"
format_stamp: "Formatted at 2026-06-20 18:43:56 on dist-test-slave-k6lg"
server_key: "5a975441a2e580069195bd617330a99c"
server_key_iv: "477f8daef0e1e858ee8ab90d00fb2a2e"
server_key_version: "kuduclusterkey@0"
I20260620 18:43:56.307505 17293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "44c41159cbe44ea58b3f885159cb95e5"
format_stamp: "Formatted at 2026-06-20 18:43:56 on dist-test-slave-k6lg"
server_key: "5a975441a2e580069195bd617330a99c"
server_key_iv: "477f8daef0e1e858ee8ab90d00fb2a2e"
server_key_version: "kuduclusterkey@0"
I20260620 18:43:56.313197 17293 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:43:56.329358 17308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:56.331095 17293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:43:56.331296 17293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "44c41159cbe44ea58b3f885159cb95e5"
format_stamp: "Formatted at 2026-06-20 18:43:56 on dist-test-slave-k6lg"
server_key: "5a975441a2e580069195bd617330a99c"
server_key_iv: "477f8daef0e1e858ee8ab90d00fb2a2e"
server_key_version: "kuduclusterkey@0"
I20260620 18:43:56.331537 17293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:56.352495 17293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:56.356669 17293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:56.356995 17293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:56.376252 17293 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:37695
I20260620 18:43:56.376302 17359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:37695 every 8 connection(s)
I20260620 18:43:56.377643 17293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:56.381029 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 17293
I20260620 18:43:56.381233 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:43:56.381893 17360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:56.382062 6022 external_mini_cluster.cc:1428] Setting key 70bd7e6b88cfaa2cbbbf974b591a83b6
I20260620 18:43:56.396174 17360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44c41159cbe44ea58b3f885159cb95e5: Bootstrap starting.
Jun 20 18:43:56 dist-test-slave-k6lg krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:43:56.400465 17360 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44c41159cbe44ea58b3f885159cb95e5: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:56.401851 17360 log.cc:824] T 00000000000000000000000000000000 P 44c41159cbe44ea58b3f885159cb95e5: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:56.404906 17360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44c41159cbe44ea58b3f885159cb95e5: No bootstrap required, opened a new log
I20260620 18:43:56.406406 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:56.384129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52276 (local address 127.5.225.190:37695)
0620 18:43:56.384998 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:56.385009 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:56.385050 (+ 41us) server_negotiation.cc:407] Connection header received
0620 18:43:56.386781 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:56.386801 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:56.387173 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:56.387500 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:56.388986 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:56.390401 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:56.392149 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:56.392848 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:56.392984 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:56.397784 (+ 4800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:56.397797 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:56.397824 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:56.397875 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:56.402310 (+ 4435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:56.403144 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:56.403155 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:56.403163 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:56.403340 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:56.403723 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:56.403729 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:56.403732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:56.404224 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:56.404510 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:56.404994 (+ 484us) server_negotiation.cc:299] Negotiation successful
0620 18:43:56.405380 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":370,"threads_started":1}
I20260620 18:43:56.412529 17360 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44c41159cbe44ea58b3f885159cb95e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c41159cbe44ea58b3f885159cb95e5" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 37695 } }
I20260620 18:43:56.412901 17360 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44c41159cbe44ea58b3f885159cb95e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:56.413004 17360 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44c41159cbe44ea58b3f885159cb95e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44c41159cbe44ea58b3f885159cb95e5, State: Initialized, Role: FOLLOWER
I20260620 18:43:56.413583 17360 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44c41159cbe44ea58b3f885159cb95e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c41159cbe44ea58b3f885159cb95e5" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 37695 } }
I20260620 18:43:56.413831 17360 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44c41159cbe44ea58b3f885159cb95e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:56.413968 17360 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44c41159cbe44ea58b3f885159cb95e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:56.414147 17360 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44c41159cbe44ea58b3f885159cb95e5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:56.416438 17360 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44c41159cbe44ea58b3f885159cb95e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c41159cbe44ea58b3f885159cb95e5" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 37695 } }
I20260620 18:43:56.416913 17360 leader_election.cc:302] T 00000000000000000000000000000000 P 44c41159cbe44ea58b3f885159cb95e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44c41159cbe44ea58b3f885159cb95e5; no voters:
I20260620 18:43:56.417501 17360 leader_election.cc:288] T 00000000000000000000000000000000 P 44c41159cbe44ea58b3f885159cb95e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:56.417634 17365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44c41159cbe44ea58b3f885159cb95e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:56.418488 17365 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44c41159cbe44ea58b3f885159cb95e5 [term 1 LEADER]: Becoming Leader. State: Replica: 44c41159cbe44ea58b3f885159cb95e5, State: Running, Role: LEADER
I20260620 18:43:56.419065 17365 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44c41159cbe44ea58b3f885159cb95e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c41159cbe44ea58b3f885159cb95e5" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 37695 } }
I20260620 18:43:56.419788 17360 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44c41159cbe44ea58b3f885159cb95e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:56.421847 17367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44c41159cbe44ea58b3f885159cb95e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44c41159cbe44ea58b3f885159cb95e5. Latest consensus state: current_term: 1 leader_uuid: "44c41159cbe44ea58b3f885159cb95e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c41159cbe44ea58b3f885159cb95e5" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 37695 } } }
I20260620 18:43:56.422300 17367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44c41159cbe44ea58b3f885159cb95e5 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:56.422745 17366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44c41159cbe44ea58b3f885159cb95e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44c41159cbe44ea58b3f885159cb95e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c41159cbe44ea58b3f885159cb95e5" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 37695 } } }
I20260620 18:43:56.423168 17366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44c41159cbe44ea58b3f885159cb95e5 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:56.424721 17370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:56.431094 17370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:56.431834 17360 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 18:43:56.457880 17370 catalog_manager.cc:1347] Generated new cluster ID: 0c53ad87cba24b6fb3ce12f463c69af1
I20260620 18:43:56.458050 17370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:56.472774 17370 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:56.475399 17370 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:56.488364 17370 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44c41159cbe44ea58b3f885159cb95e5: Generated new TSK 0
I20260620 18:43:56.489468 17370 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab.
Entry for principal kudu/127.5.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab.
I20260620 18:43:57.184497 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.225.148:33185/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.129:0
--local_ip_for_outbound_sockets=127.5.225.129
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:37695
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:36279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:43:57 dist-test-slave-k6lg krb5kdc[16849](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781981037, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 18:43:57.581715 17405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:57.582037 17405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:57.582233 17405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:57.590107 17405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:57.590274 17405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:57.590467 17405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.129
I20260620 18:43:57.602777 17405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:36279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.5.225.148:33185/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:37695
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.129
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:43:57.604633 17405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:57.605937 17405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:57.620577 17416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:57.620604 17418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:57.621414 17415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:57.621526 17405 server_base.cc:1034] running on GCE node
I20260620 18:43:57.622136 17405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:57.623098 17405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:57.624328 17405 hybrid_clock.cc:648] HybridClock initialized: now 1781981037624253 us; error 55 us; skew 500 ppm
I20260620 18:43:57.624677 17405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:43:57 dist-test-slave-k6lg krb5kdc[16849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981037, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:57.628701 17405 init.cc:377] Logged in from keytab as kudu/127.5.225.129@KRBTEST.COM (short username kudu)
I20260620 18:43:57.630612 17405 webserver.cc:460] Webserver started at http://127.5.225.129:43133/ using document root <none> and password file <none>
I20260620 18:43:57.631485 17405 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:57.631623 17405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:57.631978 17405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:43:57 dist-test-slave-k6lg krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981037, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for HTTP/127.5.225.148@KRBTEST.COM
I20260620 18:43:57.652096 17405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-0/data/instance:
uuid: "e0abdcacc32e456b885f617e5cd5e0f5"
format_stamp: "Formatted at 2026-06-20 18:43:57 on dist-test-slave-k6lg"
server_key: "04830cdd4d07080571367e83b71c4434"
server_key_iv: "9db12210b313227e72c0f0eaa51b38f6"
server_key_version: "kuduclusterkey@0"
I20260620 18:43:57.653204 17405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance:
uuid: "e0abdcacc32e456b885f617e5cd5e0f5"
format_stamp: "Formatted at 2026-06-20 18:43:57 on dist-test-slave-k6lg"
server_key: "04830cdd4d07080571367e83b71c4434"
server_key_iv: "9db12210b313227e72c0f0eaa51b38f6"
server_key_version: "kuduclusterkey@0"
I20260620 18:43:57.676678 17405 fs_manager.cc:691] Time spent creating directory manager: real 0.023s user 0.009s sys 0.016s
I20260620 18:43:57.705639 17425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:57.711649 17405 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.009s sys 0.005s
I20260620 18:43:57.711985 17405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-0/wal
uuid: "e0abdcacc32e456b885f617e5cd5e0f5"
format_stamp: "Formatted at 2026-06-20 18:43:57 on dist-test-slave-k6lg"
server_key: "04830cdd4d07080571367e83b71c4434"
server_key_iv: "9db12210b313227e72c0f0eaa51b38f6"
server_key_version: "kuduclusterkey@0"
I20260620 18:43:57.712370 17405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:57.770330 17405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:57.775871 17405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:57.776273 17405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:57.787330 17405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:57.788985 17405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:57.789100 17405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:57.789259 17405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:57.789332 17405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:57.877420 17405 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.129:45085
I20260620 18:43:57.877580 17537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.129:45085 every 8 connection(s)
I20260620 18:43:57.879150 17405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
I20260620 18:43:57.883697 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 17405
I20260620 18:43:57.883942 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance
I20260620 18:43:57.884948 6022 external_mini_cluster.cc:1428] Setting key 2ea926f7672d222f5b1c54a99d366e1e
Jun 20 18:43:57 dist-test-slave-k6lg krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981037, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:43:57.917758 17541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:57.883017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:37695 (local address 127.5.225.129:35839)
0620 18:43:57.892871 (+ 9854us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:57.892944 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:57.895621 (+ 2677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:57.898147 (+ 2526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:57.898182 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:57.898955 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:57.899894 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:57.899924 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:57.901454 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:57.901460 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:57.903552 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:57.903569 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:57.904768 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:57.904775 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:57.905058 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:57.905871 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:57.905901 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:57.908812 (+ 2911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:57.912960 (+ 4148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:57.912970 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:57.912989 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:57.913588 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:57.914014 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:57.914019 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:57.914022 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:57.914309 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:57.915017 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:57.915026 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:57.915471 (+ 445us) client_negotiation.cc:769] Sending connection context
0620 18:43:57.915778 (+ 307us) client_negotiation.cc:240] Negotiation successful
0620 18:43:57.916342 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9467,"thread_start_us":12993,"threads_started":1}
I20260620 18:43:57.918785 17540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:57.882347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:35839 (local address 127.5.225.190:37695)
0620 18:43:57.897322 (+ 14975us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:57.897329 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:57.897353 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:43:57.897419 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:57.897425 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:57.897521 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:57.897646 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:57.900117 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:57.901252 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:57.903736 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:57.904364 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:57.904480 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:57.909087 (+ 4607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:57.909094 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:57.909099 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:57.909139 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:57.912790 (+ 3651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:57.913732 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:57.913738 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:57.913742 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:57.913883 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:57.914450 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:57.914454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:57.914458 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:57.914797 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:57.914943 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:57.918196 (+ 3253us) server_negotiation.cc:299] Negotiation successful
0620 18:43:57.918430 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14825,"thread_start_us":15540,"threads_started":1}
WARNING: no policy specified for kudu/127.5.225.130@KRBTEST.COM; defaulting to no policy
I20260620 18:43:57.921273 17538 heartbeater.cc:347] Connected to a master server at 127.5.225.190:37695
I20260620 18:43:57.921636 17538 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:57.922582 17538 heartbeater.cc:511] Master 127.5.225.190:37695 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.130@KRBTEST.COM" created.
I20260620 18:43:57.925302 17325 ts_manager.cc:194] Registered new tserver with Master: e0abdcacc32e456b885f617e5cd5e0f5 (127.5.225.129:45085)
I20260620 18:43:57.927322 17325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.129@KRBTEST.COM'} at 127.5.225.129:35839
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab.
Entry for principal kudu/127.5.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab.
I20260620 18:43:57.959872 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.225.148:33185/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.130:0
--local_ip_for_outbound_sockets=127.5.225.130
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:37695
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:36279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:43:58 dist-test-slave-k6lg krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981037, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for HTTP/127.5.225.148@KRBTEST.COM
Jun 20 18:43:58 dist-test-slave-k6lg krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981037, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for HTTP/127.5.225.148@KRBTEST.COM
W20260620 18:43:58.405759 17544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:58.406131 17544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:58.406370 17544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:58.422564 17544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:58.422772 17544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:58.423007 17544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.130
I20260620 18:43:58.439391 17544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:36279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.5.225.148:33185/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:37695
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.130
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:43:58.441356 17544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:58.443076 17544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:58.480669 17551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:58.481025 17552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:58.501511 17554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:58.501706 17544 server_base.cc:1034] running on GCE node
I20260620 18:43:58.503073 17544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:58.506790 17544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:58.508037 17544 hybrid_clock.cc:648] HybridClock initialized: now 1781981038507963 us; error 66 us; skew 500 ppm
I20260620 18:43:58.508486 17544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:43:58 dist-test-slave-k6lg krb5kdc[16849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981038, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:58.512811 17544 init.cc:377] Logged in from keytab as kudu/127.5.225.130@KRBTEST.COM (short username kudu)
I20260620 18:43:58.527818 17544 webserver.cc:460] Webserver started at http://127.5.225.130:39863/ using document root <none> and password file <none>
I20260620 18:43:58.528833 17544 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:58.528985 17544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:58.529415 17544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:43:58 dist-test-slave-k6lg krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981038, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for HTTP/127.5.225.148@KRBTEST.COM
I20260620 18:43:58.548317 17544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-1/data/instance:
uuid: "e6be7d3951dc4ccbbd450e9d0b7fb9c7"
format_stamp: "Formatted at 2026-06-20 18:43:58 on dist-test-slave-k6lg"
server_key: "2b937f342ed310af1631baf23e7b1853"
server_key_iv: "c42ad032fe75112f7b9f902d319fbce3"
server_key_version: "kuduclusterkey@0"
I20260620 18:43:58.549638 17544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance:
uuid: "e6be7d3951dc4ccbbd450e9d0b7fb9c7"
format_stamp: "Formatted at 2026-06-20 18:43:58 on dist-test-slave-k6lg"
server_key: "2b937f342ed310af1631baf23e7b1853"
server_key_iv: "c42ad032fe75112f7b9f902d319fbce3"
server_key_version: "kuduclusterkey@0"
I20260620 18:43:58.576162 17544 fs_manager.cc:691] Time spent creating directory manager: real 0.026s user 0.005s sys 0.011s
I20260620 18:43:58.615746 17561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:58.617928 17544 fs_manager.cc:725] Time spent opening block manager: real 0.021s user 0.016s sys 0.004s
I20260620 18:43:58.618221 17544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-1/wal
uuid: "e6be7d3951dc4ccbbd450e9d0b7fb9c7"
format_stamp: "Formatted at 2026-06-20 18:43:58 on dist-test-slave-k6lg"
server_key: "2b937f342ed310af1631baf23e7b1853"
server_key_iv: "c42ad032fe75112f7b9f902d319fbce3"
server_key_version: "kuduclusterkey@0"
I20260620 18:43:58.618515 17544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:58.636300 17544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:58.642189 17544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:58.642589 17544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:58.645318 17544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:58.647306 17544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:58.647408 17544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:58.647534 17544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:58.647605 17544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:58.685155 17544 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.130:36413
I20260620 18:43:58.685194 17680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.130:36413 every 8 connection(s)
I20260620 18:43:58.686551 17544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
I20260620 18:43:58.691560 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 17544
I20260620 18:43:58.691720 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance
I20260620 18:43:58.692173 6022 external_mini_cluster.cc:1428] Setting key 01b9551e04f93a853c1b90d814513279
Jun 20 18:43:58 dist-test-slave-k6lg krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981038, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:43:58.707374 17683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:58.689149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:38271 (local address 127.5.225.190:37695)
0620 18:43:58.689685 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:58.689689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:58.690518 (+ 829us) server_negotiation.cc:407] Connection header received
0620 18:43:58.692531 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:58.692538 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:58.692651 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:58.692778 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:58.694189 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:58.694944 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:58.696505 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:58.696925 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:58.697059 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:58.700452 (+ 3393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:58.700458 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:58.700461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:58.700491 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:58.704043 (+ 3552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:58.704736 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:58.704745 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:58.704751 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:58.704907 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:58.705305 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:58.705312 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:58.705318 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:58.705697 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:58.705853 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:58.706578 (+ 725us) server_negotiation.cc:299] Negotiation successful
0620 18:43:58.706866 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":304,"threads_started":1}
I20260620 18:43:58.708098 17684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:58.689595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:37695 (local address 127.5.225.130:38271)
0620 18:43:58.690352 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:58.690394 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:58.692258 (+ 1864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:58.693018 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:58.693027 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:58.693456 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:58.694013 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:58.694029 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:58.695105 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:58.695111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:58.696350 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:58.696369 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:58.697151 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:58.697160 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:58.697382 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:58.698066 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:58.698093 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:58.700225 (+ 2132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:58.704186 (+ 3961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:58.704193 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:58.704207 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:58.704617 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:58.705014 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:58.705017 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:58.705019 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:58.705194 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:58.705817 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:58.705823 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:58.706136 (+ 313us) client_negotiation.cc:769] Sending connection context
0620 18:43:58.706457 (+ 321us) client_negotiation.cc:240] Negotiation successful
0620 18:43:58.706841 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":274,"threads_started":1}
I20260620 18:43:58.709695 17681 heartbeater.cc:347] Connected to a master server at 127.5.225.190:37695
I20260620 18:43:58.710014 17681 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:58.710801 17681 heartbeater.cc:511] Master 127.5.225.190:37695 requested a full tablet report, sending...
I20260620 18:43:58.712400 17325 ts_manager.cc:194] Registered new tserver with Master: e6be7d3951dc4ccbbd450e9d0b7fb9c7 (127.5.225.130:36413)
I20260620 18:43:58.713598 17325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.130@KRBTEST.COM'} at 127.5.225.130:38271
WARNING: no policy specified for kudu/127.5.225.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab.
Entry for principal kudu/127.5.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab.
I20260620 18:43:58.743480 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.225.148:33185/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.131:0
--local_ip_for_outbound_sockets=127.5.225.131
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:37695
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:36279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:43:58.932724 17538 heartbeater.cc:503] Master 127.5.225.190:37695 was elected leader, sending a full tablet report...
W20260620 18:43:59.070271 17687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:59.070538 17687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:59.070717 17687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:59.077801 17687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:59.077901 17687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:59.078094 17687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.131
I20260620 18:43:59.087687 17687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:36279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.5.225.148:33185/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:37695
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.131
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:43:59.088968 17687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:59.090267 17687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:59.102793 17692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:59.103539 17695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:59.103565 17693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:59.103901 17687 server_base.cc:1034] running on GCE node
I20260620 18:43:59.104624 17687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:59.105728 17687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:59.106974 17687 hybrid_clock.cc:648] HybridClock initialized: now 1781981039106917 us; error 73 us; skew 500 ppm
I20260620 18:43:59.107308 17687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:43:59 dist-test-slave-k6lg krb5kdc[16849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981039, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:59.110589 17687 init.cc:377] Logged in from keytab as kudu/127.5.225.131@KRBTEST.COM (short username kudu)
I20260620 18:43:59.112023 17687 webserver.cc:460] Webserver started at http://127.5.225.131:38821/ using document root <none> and password file <none>
I20260620 18:43:59.112691 17687 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:59.112840 17687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:59.113149 17687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:43:59 dist-test-slave-k6lg krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981039, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for HTTP/127.5.225.148@KRBTEST.COM
I20260620 18:43:59.129323 17687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-2/data/instance:
uuid: "e3a877c4db514461a8541807c611aab3"
format_stamp: "Formatted at 2026-06-20 18:43:59 on dist-test-slave-k6lg"
server_key: "f77e449d214b59a72fb0282f7807aa61"
server_key_iv: "6833ccc23ca535c44d555be0a9191d36"
server_key_version: "kuduclusterkey@0"
I20260620 18:43:59.130259 17687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance:
uuid: "e3a877c4db514461a8541807c611aab3"
format_stamp: "Formatted at 2026-06-20 18:43:59 on dist-test-slave-k6lg"
server_key: "f77e449d214b59a72fb0282f7807aa61"
server_key_iv: "6833ccc23ca535c44d555be0a9191d36"
server_key_version: "kuduclusterkey@0"
I20260620 18:43:59.135784 17687 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 18:43:59.151584 17702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:59.153316 17687 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 18:43:59.153607 17687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-2/wal
uuid: "e3a877c4db514461a8541807c611aab3"
format_stamp: "Formatted at 2026-06-20 18:43:59 on dist-test-slave-k6lg"
server_key: "f77e449d214b59a72fb0282f7807aa61"
server_key_iv: "6833ccc23ca535c44d555be0a9191d36"
server_key_version: "kuduclusterkey@0"
I20260620 18:43:59.153843 17687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:59.173472 17687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:59.177556 17687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:59.177857 17687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:59.178874 17687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:59.180334 17687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:59.180433 17687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:59.180583 17687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:59.180652 17687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:59.212795 17687 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.131:46047
I20260620 18:43:59.212831 17814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.131:46047 every 8 connection(s)
I20260620 18:43:59.214109 17687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
I20260620 18:43:59.217113 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 17687
I20260620 18:43:59.217311 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance
I20260620 18:43:59.217957 6022 external_mini_cluster.cc:1428] Setting key dd546eb70b61738d059a0205522d804b
Jun 20 18:43:59 dist-test-slave-k6lg krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981039, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:43:59.235505 17817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:59.216732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:38541 (local address 127.5.225.190:37695)
0620 18:43:59.217324 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:59.217331 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:59.218589 (+ 1258us) server_negotiation.cc:407] Connection header received
0620 18:43:59.220659 (+ 2070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:59.220668 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:59.220777 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:59.220873 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:59.222652 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:59.223464 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:59.225347 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:59.225832 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:59.225967 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:59.229384 (+ 3417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:59.229389 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:59.229392 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:59.229423 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:59.232129 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:59.232998 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:59.233002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:59.233004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:59.233131 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:59.233602 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:59.233608 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:59.233612 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:59.233919 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:59.234076 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:59.234903 (+ 827us) server_negotiation.cc:299] Negotiation successful
0620 18:43:59.235125 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":281,"threads_started":1}
I20260620 18:43:59.236495 17818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:59.217204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:37695 (local address 127.5.225.131:38541)
0620 18:43:59.218133 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:59.218219 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:59.220285 (+ 2066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:59.221159 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:59.221172 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:59.221723 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:59.222451 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:59.222480 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:59.223631 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:59.223638 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:59.225181 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:59.225197 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:59.226050 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:59.226059 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:59.226370 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:59.227092 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:59.227113 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:59.229141 (+ 2028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:59.232288 (+ 3147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:59.232295 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:59.232309 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:59.232874 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:59.233252 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:59.233255 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:59.233258 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:59.233472 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:59.234050 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:59.234057 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:59.234484 (+ 427us) client_negotiation.cc:769] Sending connection context
0620 18:43:59.234764 (+ 280us) client_negotiation.cc:240] Negotiation successful
0620 18:43:59.235288 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":382,"threads_started":1}
I20260620 18:43:59.237979 17815 heartbeater.cc:347] Connected to a master server at 127.5.225.190:37695
I20260620 18:43:59.238360 17815 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:59.239045 17815 heartbeater.cc:511] Master 127.5.225.190:37695 requested a full tablet report, sending...
I20260620 18:43:59.240455 17325 ts_manager.cc:194] Registered new tserver with Master: e3a877c4db514461a8541807c611aab3 (127.5.225.131:46047)
I20260620 18:43:59.241298 17325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.131@KRBTEST.COM'} at 127.5.225.131:38541
I20260620 18:43:59.246338 6022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:43:59.263695 17817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:59.250961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40312 (local address 127.5.225.190:37695)
0620 18:43:59.251112 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:59.251116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:59.251647 (+ 531us) server_negotiation.cc:407] Connection header received
0620 18:43:59.251855 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:59.251860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:59.251932 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:59.252020 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:59.253344 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:59.254059 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:59.255575 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:59.256007 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:59.256104 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:59.258592 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:59.258616 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:59.258624 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:59.258664 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:59.261255 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:59.261882 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:59.261886 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:59.261888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:59.262013 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:59.262392 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:59.262395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:59.262397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:59.262649 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:59.262866 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:59.263170 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 18:43:59.263336 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 18:43:59.267642 17325 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40312:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:43:59.274844 17325 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:43:59.312130 17831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:59.287776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58372 (local address 127.5.225.131:46047)
0620 18:43:59.302986 (+ 15210us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:59.302998 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:59.303713 (+ 715us) server_negotiation.cc:407] Connection header received
0620 18:43:59.303948 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:59.303956 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:59.304208 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:59.304367 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:59.305416 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:59.307010 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:59.309364 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:59.310899 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:59.311034 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:59.311245 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:59.311585 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 18:43:59.311766 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15026,"thread_start_us":15218,"threads_started":1}
I20260620 18:43:59.312251 17828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:59.286839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.130:36413 (local address 127.0.0.1:36900)
0620 18:43:59.302863 (+ 16024us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:59.302912 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:59.303091 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:59.304132 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:59.304139 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:59.304170 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:59.304577 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:59.304589 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:59.306380 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:59.306384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:59.308763 (+ 2379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:59.308777 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:59.311023 (+ 2246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:59.311034 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:59.311238 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:59.311383 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 18:43:59.311513 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 18:43:59.311753 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15739,"mutex_wait_us":169,"thread_start_us":15309,"threads_started":1}
I20260620 18:43:59.312275 17829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:59.287096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.131:46047 (local address 127.0.0.1:58372)
0620 18:43:59.303575 (+ 16479us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:59.303593 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:59.303736 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:59.304720 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:59.304725 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:59.304747 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:59.305225 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:59.305238 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:59.307155 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:59.307161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:59.309198 (+ 2037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:59.309213 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:59.311086 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:59.311091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:59.311349 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:59.311383 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 18:43:59.311510 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 18:43:59.311753 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16370,"mutex_wait_us":127,"thread_start_us":15486,"threads_started":1}
I20260620 18:43:59.313262 17833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:59.304749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:45085 (local address 127.0.0.1:50904)
0620 18:43:59.305375 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:59.305391 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:59.305503 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:59.306066 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:59.306070 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:59.306093 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:59.306566 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:59.306575 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:59.308401 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:59.308406 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:59.310421 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:59.310435 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:59.312493 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:59.312500 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:59.312677 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:59.312707 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 18:43:59.312800 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 18:43:59.312934 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":345,"threads_started":1}
I20260620 18:43:59.313359 17830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:59.287102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36900 (local address 127.5.225.130:36413)
0620 18:43:59.303303 (+ 16201us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:59.303309 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:59.303332 (+ 23us) server_negotiation.cc:407] Connection header received
0620 18:43:59.303408 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:59.303414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:59.303610 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:59.303857 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:59.304817 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:59.306244 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:59.308926 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:59.310559 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:59.310682 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:59.310881 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:59.312654 (+ 1773us) server_negotiation.cc:299] Negotiation successful
0620 18:43:59.313038 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16047,"thread_start_us":16129,"threads_started":1}
I20260620 18:43:59.313926 17832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:59.303498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50904 (local address 127.5.225.129:45085)
0620 18:43:59.304158 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:59.304165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:59.305499 (+ 1334us) server_negotiation.cc:407] Connection header received
0620 18:43:59.305627 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:59.305635 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:59.305879 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:59.306060 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:59.306755 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:59.308233 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:59.310594 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:59.311959 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:59.312067 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:59.312304 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:59.312865 (+ 561us) server_negotiation.cc:299] Negotiation successful
0620 18:43:59.313621 (+ 756us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":964,"threads_started":1}
I20260620 18:43:59.316834 17616 tablet_service.cc:1467] Processing CreateTablet for tablet 55672e27c80b47f0be3299894f3c8072 (DEFAULT_TABLE table=test-table [id=0304260a7c45411b9d003f7db186adcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:43:59.316835 17750 tablet_service.cc:1467] Processing CreateTablet for tablet 55672e27c80b47f0be3299894f3c8072 (DEFAULT_TABLE table=test-table [id=0304260a7c45411b9d003f7db186adcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:43:59.317097 17473 tablet_service.cc:1467] Processing CreateTablet for tablet 55672e27c80b47f0be3299894f3c8072 (DEFAULT_TABLE table=test-table [id=0304260a7c45411b9d003f7db186adcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:43:59.318300 17750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55672e27c80b47f0be3299894f3c8072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:59.318300 17616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55672e27c80b47f0be3299894f3c8072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:59.318585 17473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55672e27c80b47f0be3299894f3c8072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:59.330540 17834 tablet_bootstrap.cc:492] T 55672e27c80b47f0be3299894f3c8072 P e0abdcacc32e456b885f617e5cd5e0f5: Bootstrap starting.
I20260620 18:43:59.333043 17835 tablet_bootstrap.cc:492] T 55672e27c80b47f0be3299894f3c8072 P e6be7d3951dc4ccbbd450e9d0b7fb9c7: Bootstrap starting.
I20260620 18:43:59.334892 17834 tablet_bootstrap.cc:654] T 55672e27c80b47f0be3299894f3c8072 P e0abdcacc32e456b885f617e5cd5e0f5: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:59.334949 17836 tablet_bootstrap.cc:492] T 55672e27c80b47f0be3299894f3c8072 P e3a877c4db514461a8541807c611aab3: Bootstrap starting.
I20260620 18:43:59.336395 17834 log.cc:824] T 55672e27c80b47f0be3299894f3c8072 P e0abdcacc32e456b885f617e5cd5e0f5: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:59.336547 17835 tablet_bootstrap.cc:654] T 55672e27c80b47f0be3299894f3c8072 P e6be7d3951dc4ccbbd450e9d0b7fb9c7: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:59.337806 17835 log.cc:824] T 55672e27c80b47f0be3299894f3c8072 P e6be7d3951dc4ccbbd450e9d0b7fb9c7: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:59.338517 17836 tablet_bootstrap.cc:654] T 55672e27c80b47f0be3299894f3c8072 P e3a877c4db514461a8541807c611aab3: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:59.339794 17836 log.cc:824] T 55672e27c80b47f0be3299894f3c8072 P e3a877c4db514461a8541807c611aab3: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:59.341485 17835 tablet_bootstrap.cc:492] T 55672e27c80b47f0be3299894f3c8072 P e6be7d3951dc4ccbbd450e9d0b7fb9c7: No bootstrap required, opened a new log
I20260620 18:43:59.341719 17834 tablet_bootstrap.cc:492] T 55672e27c80b47f0be3299894f3c8072 P e0abdcacc32e456b885f617e5cd5e0f5: No bootstrap required, opened a new log
I20260620 18:43:59.341759 17835 ts_tablet_manager.cc:1397] T 55672e27c80b47f0be3299894f3c8072 P e6be7d3951dc4ccbbd450e9d0b7fb9c7: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 18:43:59.342010 17834 ts_tablet_manager.cc:1397] T 55672e27c80b47f0be3299894f3c8072 P e0abdcacc32e456b885f617e5cd5e0f5: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.003s
I20260620 18:43:59.343227 17836 tablet_bootstrap.cc:492] T 55672e27c80b47f0be3299894f3c8072 P e3a877c4db514461a8541807c611aab3: No bootstrap required, opened a new log
I20260620 18:43:59.343494 17836 ts_tablet_manager.cc:1397] T 55672e27c80b47f0be3299894f3c8072 P e3a877c4db514461a8541807c611aab3: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260620 18:43:59.348975 17835 raft_consensus.cc:348] T 55672e27c80b47f0be3299894f3c8072 P e6be7d3951dc4ccbbd450e9d0b7fb9c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6be7d3951dc4ccbbd450e9d0b7fb9c7" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 36413 } } peers { permanent_uuid: "e3a877c4db514461a8541807c611aab3" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46047 } } peers { permanent_uuid: "e0abdcacc32e456b885f617e5cd5e0f5" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 45085 } }
I20260620 18:43:59.349359 17835 raft_consensus.cc:374] T 55672e27c80b47f0be3299894f3c8072 P e6be7d3951dc4ccbbd450e9d0b7fb9c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:59.349455 17835 raft_consensus.cc:729] T 55672e27c80b47f0be3299894f3c8072 P e6be7d3951dc4ccbbd450e9d0b7fb9c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6be7d3951dc4ccbbd450e9d0b7fb9c7, State: Initialized, Role: FOLLOWER
I20260620 18:43:59.350069 17835 consensus_queue.cc:260] T 55672e27c80b47f0be3299894f3c8072 P e6be7d3951dc4ccbbd450e9d0b7fb9c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6be7d3951dc4ccbbd450e9d0b7fb9c7" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 36413 } } peers { permanent_uuid: "e3a877c4db514461a8541807c611aab3" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46047 } } peers { permanent_uuid: "e0abdcacc32e456b885f617e5cd5e0f5" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 45085 } }
I20260620 18:43:59.351158 17681 heartbeater.cc:503] Master 127.5.225.190:37695 was elected leader, sending a full tablet report...
I20260620 18:43:59.352560 17835 ts_tablet_manager.cc:1428] T 55672e27c80b47f0be3299894f3c8072 P e6be7d3951dc4ccbbd450e9d0b7fb9c7: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260620 18:43:59.354705 17834 raft_consensus.cc:348] T 55672e27c80b47f0be3299894f3c8072 P e0abdcacc32e456b885f617e5cd5e0f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6be7d3951dc4ccbbd450e9d0b7fb9c7" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 36413 } } peers { permanent_uuid: "e3a877c4db514461a8541807c611aab3" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46047 } } peers { permanent_uuid: "e0abdcacc32e456b885f617e5cd5e0f5" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 45085 } }
I20260620 18:43:59.354763 17836 raft_consensus.cc:348] T 55672e27c80b47f0be3299894f3c8072 P e3a877c4db514461a8541807c611aab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6be7d3951dc4ccbbd450e9d0b7fb9c7" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 36413 } } peers { permanent_uuid: "e3a877c4db514461a8541807c611aab3" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46047 } } peers { permanent_uuid: "e0abdcacc32e456b885f617e5cd5e0f5" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 45085 } }
I20260620 18:43:59.355173 17836 raft_consensus.cc:374] T 55672e27c80b47f0be3299894f3c8072 P e3a877c4db514461a8541807c611aab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:59.355173 17834 raft_consensus.cc:374] T 55672e27c80b47f0be3299894f3c8072 P e0abdcacc32e456b885f617e5cd5e0f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:59.355285 17836 raft_consensus.cc:729] T 55672e27c80b47f0be3299894f3c8072 P e3a877c4db514461a8541807c611aab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a877c4db514461a8541807c611aab3, State: Initialized, Role: FOLLOWER
I20260620 18:43:59.355284 17834 raft_consensus.cc:729] T 55672e27c80b47f0be3299894f3c8072 P e0abdcacc32e456b885f617e5cd5e0f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0abdcacc32e456b885f617e5cd5e0f5, State: Initialized, Role: FOLLOWER
I20260620 18:43:59.355916 17836 consensus_queue.cc:260] T 55672e27c80b47f0be3299894f3c8072 P e3a877c4db514461a8541807c611aab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6be7d3951dc4ccbbd450e9d0b7fb9c7" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 36413 } } peers { permanent_uuid: "e3a877c4db514461a8541807c611aab3" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46047 } } peers { permanent_uuid: "e0abdcacc32e456b885f617e5cd5e0f5" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 45085 } }
I20260620 18:43:59.355916 17834 consensus_queue.cc:260] T 55672e27c80b47f0be3299894f3c8072 P e0abdcacc32e456b885f617e5cd5e0f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6be7d3951dc4ccbbd450e9d0b7fb9c7" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 36413 } } peers { permanent_uuid: "e3a877c4db514461a8541807c611aab3" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46047 } } peers { permanent_uuid: "e0abdcacc32e456b885f617e5cd5e0f5" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 45085 } }
I20260620 18:43:59.357409 17815 heartbeater.cc:503] Master 127.5.225.190:37695 was elected leader, sending a full tablet report...
I20260620 18:43:59.358017 17834 ts_tablet_manager.cc:1428] T 55672e27c80b47f0be3299894f3c8072 P e0abdcacc32e456b885f617e5cd5e0f5: Time spent starting tablet: real 0.016s user 0.015s sys 0.001s
I20260620 18:43:59.358068 17836 ts_tablet_manager.cc:1428] T 55672e27c80b47f0be3299894f3c8072 P e3a877c4db514461a8541807c611aab3: Time spent starting tablet: real 0.014s user 0.007s sys 0.008s
W20260620 18:43:59.397403 17539 tablet.cc:2369] T 55672e27c80b47f0be3299894f3c8072 P e0abdcacc32e456b885f617e5cd5e0f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:43:59.438499 17682 tablet.cc:2369] T 55672e27c80b47f0be3299894f3c8072 P e6be7d3951dc4ccbbd450e9d0b7fb9c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:43:59.465710 17816 tablet.cc:2369] T 55672e27c80b47f0be3299894f3c8072 P e3a877c4db514461a8541807c611aab3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:43:59.771417 17840 raft_consensus.cc:482] T 55672e27c80b47f0be3299894f3c8072 P e6be7d3951dc4ccbbd450e9d0b7fb9c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:43:59.771823 17840 raft_consensus.cc:504] T 55672e27c80b47f0be3299894f3c8072 P e6be7d3951dc4ccbbd450e9d0b7fb9c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6be7d3951dc4ccbbd450e9d0b7fb9c7" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 36413 } } peers { permanent_uuid: "e3a877c4db514461a8541807c611aab3" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46047 } } peers { permanent_uuid: "e0abdcacc32e456b885f617e5cd5e0f5" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 45085 } }
I20260620 18:43:59.773480 17840 leader_election.cc:288] T 55672e27c80b47f0be3299894f3c8072 P e6be7d3951dc4ccbbd450e9d0b7fb9c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3a877c4db514461a8541807c611aab3 (127.5.225.131:46047), e0abdcacc32e456b885f617e5cd5e0f5 (127.5.225.129:45085)
I20260620 18:43:59.779709 17831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:59.773848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:35279 (local address 127.5.225.131:46047)
0620 18:43:59.774039 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:59.774047 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:59.774535 (+ 488us) server_negotiation.cc:407] Connection header received
0620 18:43:59.774747 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:59.774753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:59.774826 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:59.774917 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:59.775505 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:59.776189 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:59.777879 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:59.778837 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:59.779005 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:59.779078 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:59.779340 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 18:43:59.779467 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 18:43:59.779950 17843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:59.773825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.131:46047 (local address 127.5.225.130:35279)
0620 18:43:59.774424 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:59.774444 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:59.774607 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:59.774992 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:59.774999 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:59.775037 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:59.775362 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:59.775372 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:59.776326 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:59.776331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:59.777740 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:59.777753 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:59.779022 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:59.779028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:59.779185 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:59.779209 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 18:43:59.779317 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 18:43:59.779439 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":3,"thread_start_us":257,"threads_started":1}
I20260620 18:43:59.780452 17832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:59.774668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:33167 (local address 127.5.225.129:45085)
0620 18:43:59.774900 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:59.774907 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:59.775304 (+ 397us) server_negotiation.cc:407] Connection header received
0620 18:43:59.775528 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:59.775531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:59.775585 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:59.775674 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:59.776203 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:59.776903 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:59.778728 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:59.779506 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:59.779697 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:59.779766 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:59.780080 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 18:43:59.780207 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:43:59.780537 17844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:59.774501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:45085 (local address 127.5.225.130:33167)
0620 18:43:59.775199 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:59.775212 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:59.775378 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:59.775686 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:59.775690 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:59.775712 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:59.776055 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:59.776062 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:59.777027 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:59.777034 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:59.778517 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:59.778528 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:59.779724 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:59.779731 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:59.779914 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:59.779958 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 18:43:59.780055 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 18:43:59.780174 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"mutex_wait_us":36,"thread_start_us":409,"threads_started":1}
I20260620 18:43:59.780885 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55672e27c80b47f0be3299894f3c8072" candidate_uuid: "e6be7d3951dc4ccbbd450e9d0b7fb9c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a877c4db514461a8541807c611aab3" is_pre_election: true
I20260620 18:43:59.781359 17770 raft_consensus.cc:2454] T 55672e27c80b47f0be3299894f3c8072 P e3a877c4db514461a8541807c611aab3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6be7d3951dc4ccbbd450e9d0b7fb9c7 in term 0.
I20260620 18:43:59.781414 17493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55672e27c80b47f0be3299894f3c8072" candidate_uuid: "e6be7d3951dc4ccbbd450e9d0b7fb9c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0abdcacc32e456b885f617e5cd5e0f5" is_pre_election: true
I20260620 18:43:59.781890 17493 raft_consensus.cc:2454] T 55672e27c80b47f0be3299894f3c8072 P e0abdcacc32e456b885f617e5cd5e0f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6be7d3951dc4ccbbd450e9d0b7fb9c7 in term 0.
I20260620 18:43:59.781961 17565 leader_election.cc:302] T 55672e27c80b47f0be3299894f3c8072 P e6be7d3951dc4ccbbd450e9d0b7fb9c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3a877c4db514461a8541807c611aab3, e6be7d3951dc4ccbbd450e9d0b7fb9c7; no voters:
I20260620 18:43:59.782357 17840 raft_consensus.cc:2781] T 55672e27c80b47f0be3299894f3c8072 P e6be7d3951dc4ccbbd450e9d0b7fb9c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:43:59.782507 17840 raft_consensus.cc:482] T 55672e27c80b47f0be3299894f3c8072 P e6be7d3951dc4ccbbd450e9d0b7fb9c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:43:59.782621 17840 raft_consensus.cc:3037] T 55672e27c80b47f0be3299894f3c8072 P e6be7d3951dc4ccbbd450e9d0b7fb9c7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:59.784737 17840 raft_consensus.cc:504] T 55672e27c80b47f0be3299894f3c8072 P e6be7d3951dc4ccbbd450e9d0b7fb9c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6be7d3951dc4ccbbd450e9d0b7fb9c7" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 36413 } } peers { permanent_uuid: "e3a877c4db514461a8541807c611aab3" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46047 } } peers { permanent_uuid: "e0abdcacc32e456b885f617e5cd5e0f5" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 45085 } }
I20260620 18:43:59.785281 17840 leader_election.cc:288] T 55672e27c80b47f0be3299894f3c8072 P e6be7d3951dc4ccbbd450e9d0b7fb9c7 [CANDIDATE]: Term 1 election: Requested vote from peers e3a877c4db514461a8541807c611aab3 (127.5.225.131:46047), e0abdcacc32e456b885f617e5cd5e0f5 (127.5.225.129:45085)
I20260620 18:43:59.785687 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55672e27c80b47f0be3299894f3c8072" candidate_uuid: "e6be7d3951dc4ccbbd450e9d0b7fb9c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a877c4db514461a8541807c611aab3"
I20260620 18:43:59.785768 17493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55672e27c80b47f0be3299894f3c8072" candidate_uuid: "e6be7d3951dc4ccbbd450e9d0b7fb9c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0abdcacc32e456b885f617e5cd5e0f5"
I20260620 18:43:59.785928 17493 raft_consensus.cc:3037] T 55672e27c80b47f0be3299894f3c8072 P e0abdcacc32e456b885f617e5cd5e0f5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:59.785913 17770 raft_consensus.cc:3037] T 55672e27c80b47f0be3299894f3c8072 P e3a877c4db514461a8541807c611aab3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:59.789026 17493 raft_consensus.cc:2454] T 55672e27c80b47f0be3299894f3c8072 P e0abdcacc32e456b885f617e5cd5e0f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6be7d3951dc4ccbbd450e9d0b7fb9c7 in term 1.
I20260620 18:43:59.789026 17770 raft_consensus.cc:2454] T 55672e27c80b47f0be3299894f3c8072 P e3a877c4db514461a8541807c611aab3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6be7d3951dc4ccbbd450e9d0b7fb9c7 in term 1.
I20260620 18:43:59.789371 17565 leader_election.cc:302] T 55672e27c80b47f0be3299894f3c8072 P e6be7d3951dc4ccbbd450e9d0b7fb9c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3a877c4db514461a8541807c611aab3, e6be7d3951dc4ccbbd450e9d0b7fb9c7; no voters:
I20260620 18:43:59.789636 17840 raft_consensus.cc:2781] T 55672e27c80b47f0be3299894f3c8072 P e6be7d3951dc4ccbbd450e9d0b7fb9c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:59.790042 17840 raft_consensus.cc:686] T 55672e27c80b47f0be3299894f3c8072 P e6be7d3951dc4ccbbd450e9d0b7fb9c7 [term 1 LEADER]: Becoming Leader. State: Replica: e6be7d3951dc4ccbbd450e9d0b7fb9c7, State: Running, Role: LEADER
I20260620 18:43:59.790506 17840 consensus_queue.cc:237] T 55672e27c80b47f0be3299894f3c8072 P e6be7d3951dc4ccbbd450e9d0b7fb9c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6be7d3951dc4ccbbd450e9d0b7fb9c7" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 36413 } } peers { permanent_uuid: "e3a877c4db514461a8541807c611aab3" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46047 } } peers { permanent_uuid: "e0abdcacc32e456b885f617e5cd5e0f5" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 45085 } }
I20260620 18:43:59.795146 17325 catalog_manager.cc:5495] T 55672e27c80b47f0be3299894f3c8072 P e6be7d3951dc4ccbbd450e9d0b7fb9c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6be7d3951dc4ccbbd450e9d0b7fb9c7 (127.5.225.130). New cstate: current_term: 1 leader_uuid: "e6be7d3951dc4ccbbd450e9d0b7fb9c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6be7d3951dc4ccbbd450e9d0b7fb9c7" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 36413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3a877c4db514461a8541807c611aab3" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0abdcacc32e456b885f617e5cd5e0f5" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 45085 } health_report { overall_health: UNKNOWN } } }
I20260620 18:43:59.862651 17851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:59.856682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36910 (local address 127.5.225.130:36413)
0620 18:43:59.857169 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:59.857174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:59.857397 (+ 223us) server_negotiation.cc:407] Connection header received
0620 18:43:59.857636 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:59.857640 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:59.857725 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:59.857866 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:43:59.858328 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:59.859161 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:59.860595 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:59.861011 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:59.861104 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:59.861540 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:43:59.862069 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:43:59.862230 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:59.862370 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 18:43:59.862433 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":272,"threads_started":1}
I20260620 18:43:59.870374 17493 raft_consensus.cc:1261] T 55672e27c80b47f0be3299894f3c8072 P e0abdcacc32e456b885f617e5cd5e0f5 [term 1 FOLLOWER]: Refusing update from remote peer e6be7d3951dc4ccbbd450e9d0b7fb9c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:43:59.870405 17770 raft_consensus.cc:1261] T 55672e27c80b47f0be3299894f3c8072 P e3a877c4db514461a8541807c611aab3 [term 1 FOLLOWER]: Refusing update from remote peer e6be7d3951dc4ccbbd450e9d0b7fb9c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:43:59.871272 17845 consensus_queue.cc:1035] T 55672e27c80b47f0be3299894f3c8072 P e6be7d3951dc4ccbbd450e9d0b7fb9c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0abdcacc32e456b885f617e5cd5e0f5" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 45085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:43:59.871621 17840 consensus_queue.cc:1035] T 55672e27c80b47f0be3299894f3c8072 P e6be7d3951dc4ccbbd450e9d0b7fb9c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a877c4db514461a8541807c611aab3" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:43:59.883035 17856 mvcc.cc:205] Tried to move back new op lower bound from 7298994339299680256 to 7298994338985426944. Current Snapshot: MvccSnapshot[applied={T|T < 7298994339299680256}]
I20260620 18:43:59.886016 17853 mvcc.cc:205] Tried to move back new op lower bound from 7298994339299680256 to 7298994338985426944. Current Snapshot: MvccSnapshot[applied={T|T < 7298994339299680256}]
I20260620 18:43:59.900372 17324 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:43:59.900620 17324 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:43:59.903972 17324 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 44c41159cbe44ea58b3f885159cb95e5: Sending DeleteTablet for 3 replicas of tablet 55672e27c80b47f0be3299894f3c8072
I20260620 18:43:59.905085 17616 tablet_service.cc:1514] Processing DeleteTablet for tablet 55672e27c80b47f0be3299894f3c8072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:43:59 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:36900
I20260620 18:43:59.905225 17473 tablet_service.cc:1514] Processing DeleteTablet for tablet 55672e27c80b47f0be3299894f3c8072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:43:59 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:50904
I20260620 18:43:59.905913 17750 tablet_service.cc:1514] Processing DeleteTablet for tablet 55672e27c80b47f0be3299894f3c8072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:43:59 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:58372
I20260620 18:43:59.907015 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 17405
I20260620 18:43:59.919708 17867 tablet_replica.cc:330] T 55672e27c80b47f0be3299894f3c8072 P e3a877c4db514461a8541807c611aab3: stopping tablet replica
I20260620 18:43:59.920794 17866 tablet_replica.cc:330] T 55672e27c80b47f0be3299894f3c8072 P e0abdcacc32e456b885f617e5cd5e0f5: stopping tablet replica
I20260620 18:43:59.921217 17866 raft_consensus.cc:2229] T 55672e27c80b47f0be3299894f3c8072 P e0abdcacc32e456b885f617e5cd5e0f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:43:59.921711 17866 raft_consensus.cc:2258] T 55672e27c80b47f0be3299894f3c8072 P e0abdcacc32e456b885f617e5cd5e0f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:43:59.922096 17867 raft_consensus.cc:2229] T 55672e27c80b47f0be3299894f3c8072 P e3a877c4db514461a8541807c611aab3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:43:59.922602 17867 raft_consensus.cc:2258] T 55672e27c80b47f0be3299894f3c8072 P e3a877c4db514461a8541807c611aab3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:43:59.923178 17865 tablet_replica.cc:330] T 55672e27c80b47f0be3299894f3c8072 P e6be7d3951dc4ccbbd450e9d0b7fb9c7: stopping tablet replica
I20260620 18:43:59.923563 17865 raft_consensus.cc:2229] T 55672e27c80b47f0be3299894f3c8072 P e6be7d3951dc4ccbbd450e9d0b7fb9c7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:43:59.924225 17865 raft_consensus.cc:2258] T 55672e27c80b47f0be3299894f3c8072 P e6be7d3951dc4ccbbd450e9d0b7fb9c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:43:59.924669 17866 ts_tablet_manager.cc:1905] T 55672e27c80b47f0be3299894f3c8072 P e0abdcacc32e456b885f617e5cd5e0f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:43:59.926010 17867 ts_tablet_manager.cc:1905] T 55672e27c80b47f0be3299894f3c8072 P e3a877c4db514461a8541807c611aab3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:43:59.927490 17865 ts_tablet_manager.cc:1905] T 55672e27c80b47f0be3299894f3c8072 P e6be7d3951dc4ccbbd450e9d0b7fb9c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:43:59.933090 17867 ts_tablet_manager.cc:1918] T 55672e27c80b47f0be3299894f3c8072 P e3a877c4db514461a8541807c611aab3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:43:59.933290 17867 log.cc:1196] T 55672e27c80b47f0be3299894f3c8072 P e3a877c4db514461a8541807c611aab3: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-2/wal/wals/55672e27c80b47f0be3299894f3c8072
I20260620 18:43:59.933799 17867 ts_tablet_manager.cc:1939] T 55672e27c80b47f0be3299894f3c8072 P e3a877c4db514461a8541807c611aab3: Deleting consensus metadata
Jun 20 18:43:59 dist-test-slave-k6lg krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.129@KRBTEST.COM
I20260620 18:43:59.935550 17865 ts_tablet_manager.cc:1918] T 55672e27c80b47f0be3299894f3c8072 P e6be7d3951dc4ccbbd450e9d0b7fb9c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:43:59.935806 17865 log.cc:1196] T 55672e27c80b47f0be3299894f3c8072 P e6be7d3951dc4ccbbd450e9d0b7fb9c7: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-1/wal/wals/55672e27c80b47f0be3299894f3c8072
I20260620 18:43:59.935822 17312 catalog_manager.cc:4841] TS e3a877c4db514461a8541807c611aab3 (127.5.225.131:46047): tablet 55672e27c80b47f0be3299894f3c8072 (table test-table [id=0304260a7c45411b9d003f7db186adcf]) successfully deleted
I20260620 18:43:59.936360 17865 ts_tablet_manager.cc:1939] T 55672e27c80b47f0be3299894f3c8072 P e6be7d3951dc4ccbbd450e9d0b7fb9c7: Deleting consensus metadata
I20260620 18:43:59.938100 17310 catalog_manager.cc:4841] TS e6be7d3951dc4ccbbd450e9d0b7fb9c7 (127.5.225.130:36413): tablet 55672e27c80b47f0be3299894f3c8072 (table test-table [id=0304260a7c45411b9d003f7db186adcf]) successfully deleted
I20260620 18:43:59.942364 17866 ts_tablet_manager.cc:1918] T 55672e27c80b47f0be3299894f3c8072 P e0abdcacc32e456b885f617e5cd5e0f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:43:59.942521 17866 log.cc:1196] T 55672e27c80b47f0be3299894f3c8072 P e0abdcacc32e456b885f617e5cd5e0f5: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980894474392-6022-0/minicluster-data/ts-0/wal/wals/55672e27c80b47f0be3299894f3c8072
I20260620 18:43:59.943058 17866 ts_tablet_manager.cc:1939] T 55672e27c80b47f0be3299894f3c8072 P e0abdcacc32e456b885f617e5cd5e0f5: Deleting consensus metadata
I20260620 18:43:59.943873 17832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:59.908133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50914 (local address 127.5.225.129:45085)
0620 18:43:59.908304 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:59.908312 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:59.918709 (+ 10397us) server_negotiation.cc:407] Connection header received
0620 18:43:59.918928 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:59.918934 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:59.919025 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:59.919142 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:59.922071 (+ 2929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:59.923244 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:59.926565 (+ 3321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:59.927267 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:59.927379 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:59.937181 (+ 9802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:59.937201 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:59.937220 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:59.937295 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:59.940629 (+ 3334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:59.941398 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:59.941406 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:59.941408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:59.941528 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:59.941915 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:59.941920 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:59.941923 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:59.942386 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:59.942847 (+ 461us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:59.943079 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 18:43:59.943380 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:43:59.944504 17533 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:59.944762 17312 catalog_manager.cc:4841] TS e0abdcacc32e456b885f617e5cd5e0f5 (127.5.225.129:45085): tablet 55672e27c80b47f0be3299894f3c8072 (table test-table [id=0304260a7c45411b9d003f7db186adcf]) successfully deleted
I20260620 18:44:00.035161 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 17405
I20260620 18:44:00.052697 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 17544
Jun 20 18:44:00 dist-test-slave-k6lg krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.130@KRBTEST.COM
I20260620 18:44:00.071606 17851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:00.053635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36920 (local address 127.5.225.130:36413)
0620 18:44:00.053848 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:00.053856 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:00.053880 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:44:00.053950 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:00.053956 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:00.054040 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:00.054180 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:00.055519 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:00.056716 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:00.058293 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:00.059072 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:00.059209 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:00.063609 (+ 4400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:00.063630 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:00.063653 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:00.063698 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:00.068190 (+ 4492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:00.068889 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:00.068899 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:00.068903 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:00.069079 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:00.069534 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:00.069540 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:00.069545 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:00.070016 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:00.070407 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:00.070833 (+ 426us) server_negotiation.cc:299] Negotiation successful
0620 18:44:00.071149 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 18:44:00.072404 17676 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:00.185039 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 17544
I20260620 18:44:00.203063 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 17687
Jun 20 18:44:00 dist-test-slave-k6lg krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.131@KRBTEST.COM
I20260620 18:44:00.219619 17831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:00.203979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58386 (local address 127.5.225.131:46047)
0620 18:44:00.204137 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:00.204141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:00.204159 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:44:00.204233 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:00.204236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:00.204296 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:00.204386 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:00.205875 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:00.206709 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:00.208138 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:00.208540 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:00.208643 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:00.213613 (+ 4970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:00.213624 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:00.213637 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:00.213671 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:00.216668 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:00.217454 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:00.217461 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:00.217463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:00.217586 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:00.217961 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:00.217967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:00.217970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:00.218388 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:00.218662 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:00.218978 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 18:44:00.219203 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 18:44:00.220356 17810 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:00.309089 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 17687
I20260620 18:44:00.325160 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 17293
I20260620 18:44:00.326378 17355 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:00.438125 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 17293
I20260620 18:44:00.461645 6022 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 18:44:00.796679 6022 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 18:44:00.796909 6022 mini_ranger.cc:66] Stopping Ranger...
I20260620 18:44:01.142086 6022 mini_ranger.cc:68] Stopped Ranger
2026-06-20 18:44:01.142 UTC [16879] LOG: received smart shutdown request
2026-06-20 18:44:01.145 UTC [16879] LOG: background worker "logical replication launcher" (PID 16888) exited with exit code 1
2026-06-20 18:44:01.145 UTC [16883] LOG: shutting down
2026-06-20 18:44:01.292 UTC [16879] LOG: database system is shut down
2026-06-20T18:44:01Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48375 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:44:01 dist-test-slave-k6lg krb5kdc[17881](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:44:01 dist-test-slave-k6lg krb5kdc[17881](info): set up 2 sockets
Jun 20 18:44:01 dist-test-slave-k6lg krb5kdc[17881](info): commencing operation
krb5kdc: starting...
W20260620 18:44:02.534688 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:44:02 dist-test-slave-k6lg krb5kdc[17881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:44:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:44:02Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.5.225.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.5.225.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.5.225.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/rangeradmin_127.5.225.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.5.225.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/rangeradmin_127.5.225.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.5.225.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.5.225.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.5.225.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/rangerlookup_127.5.225.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.5.225.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/rangerlookup_127.5.225.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.225.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.225.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.225.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/HTTP_127.5.225.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.225.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/HTTP_127.5.225.148@KRBTEST.COM.keytab.
I20260620 18:44:02.847290 6022 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskuQUw40/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/postgres -l logfile start
2026-06-20 18:44:05.254 UTC [17911] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 18:44:05.254 UTC [17911] LOG: listening on IPv4 address "127.5.225.148", port 45875
2026-06-20 18:44:05.257 UTC [17911] LOG: listening on Unix socket "/tmp/.s.PGSQL.45875"
2026-06-20 18:44:05.279 UTC [17914] LOG: database system was shut down at 2026-06-20 18:44:03 UTC
2026-06-20 18:44:05.283 UTC [17911] LOG: database system is ready to accept connections
I20260620 18:44:06.282290 6022 mini_postgres.cc:96] Postgres bound to 45875
2026-06-20 18:44:06.303 UTC [17922] FATAL: database "slave" does not exist
127.5.225.148:45875 - accepting connections
I20260620 18:44:06.304132 6022 mini_ranger.cc:161] Starting Ranger...
I20260620 18:44:06.330797 6022 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 18:44:06.811769 6022 mini_ranger.cc:87] Created ranger Postgres database
I20260620 18:44:06.811890 6022 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ranger-admin
2026-06-20 18:44:07,089 [I] DB FLAVOR :POSTGRES
2026-06-20 18:44:07,089 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 18:44:07,089 [I] Checking connection..
2026-06-20 18:44:07,089 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:45875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 18:44:07,353 [I] Checking connection passed.
2026-06-20 18:44:07,354 [I] --------- Verifying version history table ---------
2026-06-20 18:44:07,354 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:45875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:44:07,620 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 18:44:07,620 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:45875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:44:07,848 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 18:44:07,849 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 18:44:07,849 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:45875/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 18:44:08.083 UTC [17986] WARNING: there is no transaction in progress
2026-06-20 18:44:08,089 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 18:44:08,090 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:45875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:44:08,330 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 18:44:08,330 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 18:44:08,330 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:45875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 18:44:08,592 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:45875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 18:44:08,849 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:45875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-k6lg.c.gcp-upstream.internal','N') ;"
2026-06-20 18:44:09,084 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:45875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 18:44:09,334 [I] Table x_portal_user does not exist in database ranger
2026-06-20 18:44:09,334 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:45875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 18:44:09,580 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 18:44:09,580 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:45875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 18:44:09,820 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:45875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 18:44:10,081 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 18:44:10,081 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:45875/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 18:44:11.011 UTC [18094] WARNING: there is no transaction in progress
2026-06-20 18:44:11.027 UTC [18094] WARNING: there is no transaction in progress
2026-06-20 18:44:11.043 UTC [18094] WARNING: there is no transaction in progress
2026-06-20 18:44:11.059 UTC [18094] WARNING: there is no transaction in progress
2026-06-20 18:44:11.074 UTC [18094] WARNING: there is no transaction in progress
2026-06-20 18:44:11.091 UTC [18094] WARNING: there is no transaction in progress
2026-06-20 18:44:11.145 UTC [18094] WARNING: there is no transaction in progress
2026-06-20 18:44:11.152 UTC [18094] WARNING: there is no transaction in progress
2026-06-20 18:44:11.166 UTC [18094] WARNING: there is no transaction in progress
2026-06-20 18:44:11.176 UTC [18094] WARNING: there is no transaction in progress
2026-06-20 18:44:11.185 UTC [18094] WARNING: there is no transaction in progress
2026-06-20 18:44:11.198 UTC [18094] WARNING: there is no transaction in progress
2026-06-20 18:44:11.207 UTC [18094] WARNING: there is no transaction in progress
2026-06-20 18:44:11.213 UTC [18094] WARNING: there is no transaction in progress
2026-06-20 18:44:11.219 UTC [18094] WARNING: there is no transaction in progress
2026-06-20 18:44:11.225 UTC [18094] WARNING: there is no transaction in progress
2026-06-20 18:44:11,877 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 18:44:11,877 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:45875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 18:44:12,118 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 18:44:12,119 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:45875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 18:44:12,379 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 18:44:12,380 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:45875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 18:44:12,626 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 18:44:12,626 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:45875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 18:44:12,885 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:45875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 18:44:13,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:45875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-k6lg.c.gcp-upstream.internal';"
2026-06-20 18:44:13,391 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 18:44:13,392 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:45875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 18:44:13,628 [I] DB_PATCHES have already been applied
I20260620 18:44:13.634004 6022 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ranger-admin:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskuQUw40/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 18:44:13.634093 6022 mini_ranger.cc:193] Using host: 127.5.225.148
I20260620 18:44:13.648624 6022 mini_ranger.cc:227] Ranger admin URL: http://127.5.225.148:42613
Jun 20, 2026 6:44:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:44:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:44:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 6:44:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 6:44:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 6:44:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 6:44:14 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 18:44:14.786955 6022 mini_ranger.cc:160] Time spent starting Ranger: real 8.483s user 0.000s sys 0.037s
Jun 20, 2026 6:44:34 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 6:44:34 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 6:44:34 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 6:44:34 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 6:44:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 6:44:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 6:44:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 6:44:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 6:44:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 6:44:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 6:44:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 6:44:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 6:44:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 6:44:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 6:44:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 6:44:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 6:44:34 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 6:44:35 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 18:44:36.391450 6022 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.5.225.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.5.225.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.5.225.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/rangerkms_127.5.225.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.5.225.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/rangerkms_127.5.225.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.225.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.5.225.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.225.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/HTTP_127.5.225.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.225.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/HTTP_127.5.225.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 18:44:36 dist-test-slave-k6lg krb5kdc[17881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981076, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 18:44:36.555898 6022 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 18:44:36.575593 6022 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 18:44:37.462723 6022 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 18:44:37.462854 6022 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ranger-kms
I20260620 18:44:37.462906 6022 mini_ranger_kms.cc:227] Using postgres at 127.5.225.148:45875
2026-06-20 18:44:37,816 [I] DB FLAVOR :POSTGRES
2026-06-20 18:44:37,817 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 18:44:37,817 [I] Checking connection
2026-06-20 18:44:37,817 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:45875/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 18:44:38,054 [I] connection success
2026-06-20 18:44:38,055 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 18:44:38,055 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 18:44:38,055 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:45875/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 18:44:38,290 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 18:44:38,290 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 18:44:38,290 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 18:44:38,290 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.225.148:45875/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 18:44:38,557 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 18:44:38.999400 6022 mini_ranger_kms.cc:310] Created kms service
I20260620 18:44:39.187276 6022 mini_ranger_kms.cc:326] Created kudu user
I20260620 18:44:39.231097 6022 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 18:44:39 dist-test-slave-k6lg krb5kdc[17881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981076, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.5.225.148@KRBTEST.COM
I20260620 18:44:39.648370 6022 mini_ranger_kms.cc:384] Added ranger policy
I20260620 18:44:39.648736 6022 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ranger-kms:/tmp/dist-test-taskuQUw40/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskuQUw40/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 18:44:39.648805 6022 mini_ranger_kms.cc:240] Using host: 127.5.225.148
I20260620 18:44:39.663502 6022 mini_ranger_kms.cc:276] Ranger KMS PID: 18274
I20260620 18:44:39.663626 6022 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.5.225.148:58191
Jun 20, 2026 6:44:40 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:44:40 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:44:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 6:44:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 6:44:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 6:44:41 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 18:44:41.946301 6022 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.390s user 0.096s sys 0.265s
I20260620 18:44:41.946537 6022 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 18:44:41.946710 6022 mini_ranger_kms.cc:401] 127.5.225.148:58191/kms/v1/keys
Jun 20 18:44:42 dist-test-slave-k6lg krb5kdc[17881](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781981082, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.225.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 18:44:42 dist-test-slave-k6lg krb5kdc[17881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981082, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.225.148@KRBTEST.COM for HTTP/127.5.225.148@KRBTEST.COM
Jun 20 18:44:42 dist-test-slave-k6lg krb5kdc[17881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981082, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.225.148@KRBTEST.COM for HTTP/127.5.225.148@KRBTEST.COM
Jun 20 18:44:45 dist-test-slave-k6lg krb5kdc[17881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.5.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
I20260620 18:44:45.355324 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:38707
--webserver_interface=127.5.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:35267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:38707
--ranger_config_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.225.148:58191/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:45.709371 18325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:45.709703 18325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:45.709905 18325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:45.717173 18325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:44:45.717335 18325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:45.717442 18325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:45.717542 18325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:44:45.717633 18325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:44:45.727883 18325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:35267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/master-0/wal
--ranger_kms_url=127.5.225.148:58191/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:38707
--ranger_config_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:38707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:44:45.729221 18325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:45.730702 18325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:45.744493 18331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:45.744688 18330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:45.745100 18333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:45.745107 18325 server_base.cc:1034] running on GCE node
I20260620 18:44:45.746155 18325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:45.747956 18325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:45.749236 18325 hybrid_clock.cc:648] HybridClock initialized: now 1781981085749187 us; error 67 us; skew 500 ppm
I20260620 18:44:45.749630 18325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:45 dist-test-slave-k6lg krb5kdc[17881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981085, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:45.753302 18325 init.cc:377] Logged in from keytab as kudu/127.5.225.190@KRBTEST.COM (short username kudu)
I20260620 18:44:45.755004 18325 webserver.cc:460] Webserver started at http://127.5.225.190:34479/ using document root <none> and password file <none>
I20260620 18:44:45.755682 18325 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:45.755779 18325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:45.756129 18325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:44:45 dist-test-slave-k6lg krb5kdc[17881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981085, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for HTTP/127.5.225.148@KRBTEST.COM
I20260620 18:44:45.805372 18325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "e20d976f1e114be4a9476558d845c666"
format_stamp: "Formatted at 2026-06-20 18:44:45 on dist-test-slave-k6lg"
server_key: "6645d28bf18c29bff321460c94665021"
server_key_iv: "096bffbf0e483802719607d1cc9d951f"
server_key_version: "kuduclusterkey@0"
I20260620 18:44:45.806306 18325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "e20d976f1e114be4a9476558d845c666"
format_stamp: "Formatted at 2026-06-20 18:44:45 on dist-test-slave-k6lg"
server_key: "6645d28bf18c29bff321460c94665021"
server_key_iv: "096bffbf0e483802719607d1cc9d951f"
server_key_version: "kuduclusterkey@0"
I20260620 18:44:45.811599 18325 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:44:45.827435 18340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:45.829008 18325 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:44:45.829183 18325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "e20d976f1e114be4a9476558d845c666"
format_stamp: "Formatted at 2026-06-20 18:44:45 on dist-test-slave-k6lg"
server_key: "6645d28bf18c29bff321460c94665021"
server_key_iv: "096bffbf0e483802719607d1cc9d951f"
server_key_version: "kuduclusterkey@0"
I20260620 18:44:45.829424 18325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:45.864899 18325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:45.869485 18325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:45.869830 18325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:45.889510 18325 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:38707
I20260620 18:44:45.889510 18391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:38707 every 8 connection(s)
I20260620 18:44:45.891083 18325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:44:45.893235 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 18325
I20260620 18:44:45.893468 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:44:45.894341 6022 external_mini_cluster.cc:1428] Setting key 4c6ff8a1dba60395d90b6c26be4c7a0b
I20260620 18:44:45.895280 18392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:45.913374 18392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666: Bootstrap starting.
I20260620 18:44:45.916885 18392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:45.918026 18392 log.cc:824] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666: Log is configured to *not* fsync() on all Append() calls
Jun 20 18:44:45 dist-test-slave-k6lg krb5kdc[17881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:44:45.921108 18392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666: No bootstrap required, opened a new log
I20260620 18:44:45.930310 18395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:45.896471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45150 (local address 127.5.225.190:38707)
0620 18:44:45.906744 (+ 10273us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:45.906764 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:45.907154 (+ 390us) server_negotiation.cc:407] Connection header received
0620 18:44:45.909239 (+ 2085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:45.909273 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:45.909843 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:45.910340 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:45.911831 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:45.913763 (+ 1932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:45.915622 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:45.916613 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:45.916809 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:45.921612 (+ 4803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:45.921622 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:45.921640 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:45.921685 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:45.925432 (+ 3747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:45.926325 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:45.926333 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:45.926341 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:45.926534 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:45.926987 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:45.926992 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:45.926995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:45.927773 (+ 778us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:45.928092 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:45.928550 (+ 458us) server_negotiation.cc:299] Negotiation successful
0620 18:44:45.928963 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10003,"thread_start_us":9794,"threads_started":1}
I20260620 18:44:45.930063 18392 raft_consensus.cc:348] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e20d976f1e114be4a9476558d845c666" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 38707 } }
I20260620 18:44:45.930510 18392 raft_consensus.cc:374] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:45.930629 18392 raft_consensus.cc:729] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e20d976f1e114be4a9476558d845c666, State: Initialized, Role: FOLLOWER
I20260620 18:44:45.931188 18392 consensus_queue.cc:260] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e20d976f1e114be4a9476558d845c666" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 38707 } }
I20260620 18:44:45.931443 18392 raft_consensus.cc:388] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:44:45.931634 18392 raft_consensus.cc:482] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:44:45.931895 18392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:45.934808 18392 raft_consensus.cc:504] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e20d976f1e114be4a9476558d845c666" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 38707 } }
I20260620 18:44:45.935345 18392 leader_election.cc:302] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e20d976f1e114be4a9476558d845c666; no voters:
I20260620 18:44:45.935925 18392 leader_election.cc:288] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:44:45.936053 18397 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:44:45.939195 18392 sys_catalog.cc:564] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:44:45.943168 18397 raft_consensus.cc:686] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666 [term 1 LEADER]: Becoming Leader. State: Replica: e20d976f1e114be4a9476558d845c666, State: Running, Role: LEADER
I20260620 18:44:45.943782 18397 consensus_queue.cc:237] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e20d976f1e114be4a9476558d845c666" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 38707 } }
I20260620 18:44:45.946558 18398 sys_catalog.cc:455] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e20d976f1e114be4a9476558d845c666" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e20d976f1e114be4a9476558d845c666" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 38707 } } }
I20260620 18:44:45.946913 18398 sys_catalog.cc:458] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666 [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:45.947489 18399 sys_catalog.cc:455] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e20d976f1e114be4a9476558d845c666. Latest consensus state: current_term: 1 leader_uuid: "e20d976f1e114be4a9476558d845c666" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e20d976f1e114be4a9476558d845c666" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 38707 } } }
I20260620 18:44:45.947837 18399 sys_catalog.cc:458] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666 [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:45.959741 18405 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:44:45.966202 18405 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:44:45.966876 18392 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 18:44:45.989019 18405 catalog_manager.cc:1347] Generated new cluster ID: 1944c2fbb2bd48dc8a44fe086b9ea4b1
I20260620 18:44:45.989281 18405 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:44:46.016996 18405 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:44:46.019197 18405 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:44:46.029855 18405 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666: Generated new TSK 0
I20260620 18:44:46.031148 18405 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab.
Entry for principal kudu/127.5.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab.
I20260620 18:44:46.790592 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.225.148:58191/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.129:0
--local_ip_for_outbound_sockets=127.5.225.129
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:38707
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:35267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:44:47 dist-test-slave-k6lg krb5kdc[17881](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781981087, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 18:44:47.212141 18438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:47.212625 18438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:47.212836 18438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:47.224725 18438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:47.224933 18438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:47.225152 18438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.129
I20260620 18:44:47.239933 18438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:35267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.5.225.148:58191/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:38707
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.129
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:44:47.241629 18438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:47.243419 18438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:47.260111 18449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:47.261308 18448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:47.262022 18451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:47.264040 18438 server_base.cc:1034] running on GCE node
I20260620 18:44:47.264842 18438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:47.267490 18438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:47.268795 18438 hybrid_clock.cc:648] HybridClock initialized: now 1781981087268727 us; error 61 us; skew 500 ppm
I20260620 18:44:47.269263 18438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:47 dist-test-slave-k6lg krb5kdc[17881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981087, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:47.273748 18438 init.cc:377] Logged in from keytab as kudu/127.5.225.129@KRBTEST.COM (short username kudu)
I20260620 18:44:47.275974 18438 webserver.cc:460] Webserver started at http://127.5.225.129:37235/ using document root <none> and password file <none>
I20260620 18:44:47.277026 18438 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:47.277179 18438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:47.277566 18438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:44:47 dist-test-slave-k6lg krb5kdc[17881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981087, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for HTTP/127.5.225.148@KRBTEST.COM
I20260620 18:44:47.305572 18438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-0/data/instance:
uuid: "edc7c06ba14b4d049cc3ac674929e12b"
format_stamp: "Formatted at 2026-06-20 18:44:47 on dist-test-slave-k6lg"
server_key: "59194f8ee02762d0cf263b021bae91cf"
server_key_iv: "a744ba2cc8ad2b85f7028e821d3408c0"
server_key_version: "kuduclusterkey@0"
I20260620 18:44:47.306902 18438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance:
uuid: "edc7c06ba14b4d049cc3ac674929e12b"
format_stamp: "Formatted at 2026-06-20 18:44:47 on dist-test-slave-k6lg"
server_key: "59194f8ee02762d0cf263b021bae91cf"
server_key_iv: "a744ba2cc8ad2b85f7028e821d3408c0"
server_key_version: "kuduclusterkey@0"
I20260620 18:44:47.317174 18438 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.008s sys 0.000s
I20260620 18:44:47.344106 18458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:47.346249 18438 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.005s sys 0.000s
I20260620 18:44:47.346535 18438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-0/wal
uuid: "edc7c06ba14b4d049cc3ac674929e12b"
format_stamp: "Formatted at 2026-06-20 18:44:47 on dist-test-slave-k6lg"
server_key: "59194f8ee02762d0cf263b021bae91cf"
server_key_iv: "a744ba2cc8ad2b85f7028e821d3408c0"
server_key_version: "kuduclusterkey@0"
I20260620 18:44:47.346923 18438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:47.376223 18438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:47.381541 18438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:47.381856 18438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:47.384215 18438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:44:47.386229 18438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:44:47.386399 18438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 18:44:47.386535 18438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:44:47.386595 18438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:47.455611 18438 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.129:41987
I20260620 18:44:47.455633 18570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.129:41987 every 8 connection(s)
I20260620 18:44:47.457399 18438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
I20260620 18:44:47.465974 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 18438
I20260620 18:44:47.466344 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance
I20260620 18:44:47.467201 6022 external_mini_cluster.cc:1428] Setting key 733365a4ca0d48fae50c11283184bbe5
Jun 20 18:44:47 dist-test-slave-k6lg krb5kdc[17881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981087, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:44:47.493844 18573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:47.461073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:38707 (local address 127.5.225.129:44515)
0620 18:44:47.461924 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:47.461974 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:47.464466 (+ 2492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:47.469325 (+ 4859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:47.469338 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:47.470141 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:47.471088 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:47.471113 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:47.472652 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:47.472657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:47.474790 (+ 2133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:47.474820 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:47.476128 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:47.476134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:47.476421 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:47.481509 (+ 5088us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:47.481544 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:47.484517 (+ 2973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:47.488757 (+ 4240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:47.488766 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:47.488786 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:47.489412 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:47.489900 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:47.489948 (+ 48us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:47.489952 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:47.490241 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:47.491004 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:47.491013 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:47.491469 (+ 456us) client_negotiation.cc:769] Sending connection context
0620 18:44:47.491798 (+ 329us) client_negotiation.cc:240] Negotiation successful
0620 18:44:47.492368 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":2132,"threads_started":1}
I20260620 18:44:47.495019 18574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:47.463181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:44515 (local address 127.5.225.190:38707)
0620 18:44:47.466664 (+ 3483us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:47.466671 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:47.466694 (+ 23us) server_negotiation.cc:407] Connection header received
0620 18:44:47.466759 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:47.466765 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:47.466850 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:47.466976 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:47.471301 (+ 4325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:47.472474 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:47.474995 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:47.475666 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:47.475766 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:47.484816 (+ 9050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:47.484824 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:47.484829 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:47.484872 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:47.488572 (+ 3700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:47.489573 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:47.489579 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:47.489583 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:47.489756 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:47.490383 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:47.490388 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:47.490391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:47.490751 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:47.490900 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:47.494426 (+ 3526us) server_negotiation.cc:299] Negotiation successful
0620 18:44:47.494644 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3341,"thread_start_us":3220,"threads_started":1}
I20260620 18:44:47.497386 18571 heartbeater.cc:347] Connected to a master server at 127.5.225.190:38707
I20260620 18:44:47.497726 18571 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:47.498608 18571 heartbeater.cc:511] Master 127.5.225.190:38707 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.225.130@KRBTEST.COM; defaulting to no policy
I20260620 18:44:47.503094 18357 ts_manager.cc:194] Registered new tserver with Master: edc7c06ba14b4d049cc3ac674929e12b (127.5.225.129:41987)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.130@KRBTEST.COM" created.
I20260620 18:44:47.505203 18357 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.129@KRBTEST.COM'} at 127.5.225.129:44515
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab.
Entry for principal kudu/127.5.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab.
I20260620 18:44:47.539026 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.225.148:58191/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.130:0
--local_ip_for_outbound_sockets=127.5.225.130
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:38707
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:35267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:44:47 dist-test-slave-k6lg krb5kdc[17881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981087, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for HTTP/127.5.225.148@KRBTEST.COM
Jun 20 18:44:47 dist-test-slave-k6lg krb5kdc[17881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981087, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for HTTP/127.5.225.148@KRBTEST.COM
W20260620 18:44:47.969563 18577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:47.970103 18577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:47.970357 18577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:47.982687 18577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:47.982930 18577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:47.983212 18577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.130
I20260620 18:44:47.999428 18577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:35267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.5.225.148:58191/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:38707
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.130
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:44:48.001309 18577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:48.003213 18577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:48.025027 18585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:48.025496 18584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:48.044737 18588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:48.045859 18577 server_base.cc:1034] running on GCE node
I20260620 18:44:48.046733 18577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:48.062481 18577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:48.063858 18577 hybrid_clock.cc:648] HybridClock initialized: now 1781981088063736 us; error 133 us; skew 500 ppm
I20260620 18:44:48.064360 18577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:48 dist-test-slave-k6lg krb5kdc[17881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981088, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:48.068897 18577 init.cc:377] Logged in from keytab as kudu/127.5.225.130@KRBTEST.COM (short username kudu)
I20260620 18:44:48.102285 18577 webserver.cc:460] Webserver started at http://127.5.225.130:33301/ using document root <none> and password file <none>
I20260620 18:44:48.103513 18577 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:48.103736 18577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:48.104176 18577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:44:48 dist-test-slave-k6lg krb5kdc[17881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981088, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for HTTP/127.5.225.148@KRBTEST.COM
I20260620 18:44:48.126765 18577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-1/data/instance:
uuid: "0920e736e9744560bf882d5069c9e91a"
format_stamp: "Formatted at 2026-06-20 18:44:48 on dist-test-slave-k6lg"
server_key: "f1b3e9da0fed3a8e1a98626a4bc697c2"
server_key_iv: "eb6b35c77776ef47f79dfddeffc55ead"
server_key_version: "kuduclusterkey@0"
I20260620 18:44:48.127961 18577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance:
uuid: "0920e736e9744560bf882d5069c9e91a"
format_stamp: "Formatted at 2026-06-20 18:44:48 on dist-test-slave-k6lg"
server_key: "f1b3e9da0fed3a8e1a98626a4bc697c2"
server_key_iv: "eb6b35c77776ef47f79dfddeffc55ead"
server_key_version: "kuduclusterkey@0"
I20260620 18:44:48.141279 18577 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.007s sys 0.004s
I20260620 18:44:48.190874 18594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:48.195729 18577 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.007s sys 0.008s
I20260620 18:44:48.196051 18577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-1/wal
uuid: "0920e736e9744560bf882d5069c9e91a"
format_stamp: "Formatted at 2026-06-20 18:44:48 on dist-test-slave-k6lg"
server_key: "f1b3e9da0fed3a8e1a98626a4bc697c2"
server_key_iv: "eb6b35c77776ef47f79dfddeffc55ead"
server_key_version: "kuduclusterkey@0"
I20260620 18:44:48.196391 18577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:48.267661 18577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:48.272972 18577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:48.273351 18577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:48.280791 18577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:44:48.303194 18577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:44:48.303395 18577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 18:44:48.303565 18577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:44:48.303666 18577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:48.410027 18577 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.130:35497
I20260620 18:44:48.410050 18713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.130:35497 every 8 connection(s)
I20260620 18:44:48.411653 18577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
I20260620 18:44:48.416656 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 18577
I20260620 18:44:48.416857 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance
I20260620 18:44:48.417527 6022 external_mini_cluster.cc:1428] Setting key db99c3f025c710a430b2484061ecbde8
Jun 20 18:44:48 dist-test-slave-k6lg krb5kdc[17881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981088, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.225.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.131@KRBTEST.COM" created.
I20260620 18:44:48.463188 18717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:48.418436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:35707 (local address 127.5.225.190:38707)
0620 18:44:48.419252 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:48.419258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:48.430214 (+ 10956us) server_negotiation.cc:407] Connection header received
0620 18:44:48.434342 (+ 4128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:48.434351 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:48.434467 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:48.434555 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:48.436949 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:48.438249 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:48.441047 (+ 2798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:48.441830 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:48.441949 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:48.446999 (+ 5050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:48.447005 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:48.447010 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:48.447056 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:48.450473 (+ 3417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:48.451663 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:48.451668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:48.451671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:48.451825 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:48.460588 (+ 8763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:48.460596 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:48.460602 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:48.461079 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:48.461236 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:48.462381 (+ 1145us) server_negotiation.cc:299] Negotiation successful
0620 18:44:48.462666 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":1570,"threads_started":1}
I20260620 18:44:48.465859 18716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:48.415642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:38707 (local address 127.5.225.130:35707)
0620 18:44:48.428280 (+ 12638us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:48.428335 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:48.431110 (+ 2775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:48.435159 (+ 4049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:48.435174 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:48.435864 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:48.436720 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:48.436762 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:48.438455 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:48.438462 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:48.440727 (+ 2265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:48.440765 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:48.442355 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:48.442365 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:48.442701 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:48.443573 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:48.443611 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:48.446757 (+ 3146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:48.450881 (+ 4124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:48.450892 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:48.450909 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:48.451464 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:48.451976 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:48.451979 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:48.451982 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:48.460392 (+ 8410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:48.461444 (+ 1052us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:48.461455 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:48.461926 (+ 471us) client_negotiation.cc:769] Sending connection context
0620 18:44:48.463789 (+ 1863us) client_negotiation.cc:240] Negotiation successful
0620 18:44:48.464391 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12251,"thread_start_us":15647,"threads_started":1}
I20260620 18:44:48.467708 18714 heartbeater.cc:347] Connected to a master server at 127.5.225.190:38707
I20260620 18:44:48.468022 18714 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:48.468885 18714 heartbeater.cc:511] Master 127.5.225.190:38707 requested a full tablet report, sending...
I20260620 18:44:48.470510 18357 ts_manager.cc:194] Registered new tserver with Master: 0920e736e9744560bf882d5069c9e91a (127.5.225.130:35497)
I20260620 18:44:48.471763 18357 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.130@KRBTEST.COM'} at 127.5.225.130:35707
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab.
Entry for principal kudu/127.5.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab.
I20260620 18:44:48.487756 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.225.148:58191/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.131:0
--local_ip_for_outbound_sockets=127.5.225.131
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:38707
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:35267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:44:48.509119 18571 heartbeater.cc:503] Master 127.5.225.190:38707 was elected leader, sending a full tablet report...
W20260620 18:44:48.815533 18720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:48.815841 18720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:48.816000 18720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:48.823374 18720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:48.823513 18720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:48.823675 18720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.131
I20260620 18:44:48.833495 18720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:35267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.5.225.148:58191/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:38707
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.131
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:44:48.834942 18720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:48.836215 18720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:48.849373 18725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:48.849357 18726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:48.849786 18728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:48.850127 18720 server_base.cc:1034] running on GCE node
I20260620 18:44:48.850749 18720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:48.851750 18720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:48.852979 18720 hybrid_clock.cc:648] HybridClock initialized: now 1781981088852907 us; error 61 us; skew 500 ppm
I20260620 18:44:48.853358 18720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:48 dist-test-slave-k6lg krb5kdc[17881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981088, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:48.856767 18720 init.cc:377] Logged in from keytab as kudu/127.5.225.131@KRBTEST.COM (short username kudu)
I20260620 18:44:48.858389 18720 webserver.cc:460] Webserver started at http://127.5.225.131:37855/ using document root <none> and password file <none>
I20260620 18:44:48.859102 18720 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:48.859242 18720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:48.859555 18720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:44:48 dist-test-slave-k6lg krb5kdc[17881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981088, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for HTTP/127.5.225.148@KRBTEST.COM
I20260620 18:44:48.877132 18720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-2/data/instance:
uuid: "c8dcaff0cf3c425bab8661e8ba7c3424"
format_stamp: "Formatted at 2026-06-20 18:44:48 on dist-test-slave-k6lg"
server_key: "654463af55929cbeb950555b5d3434ef"
server_key_iv: "847af6b94dde307d6cfb58492e25c4ba"
server_key_version: "kuduclusterkey@0"
I20260620 18:44:48.878104 18720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance:
uuid: "c8dcaff0cf3c425bab8661e8ba7c3424"
format_stamp: "Formatted at 2026-06-20 18:44:48 on dist-test-slave-k6lg"
server_key: "654463af55929cbeb950555b5d3434ef"
server_key_iv: "847af6b94dde307d6cfb58492e25c4ba"
server_key_version: "kuduclusterkey@0"
I20260620 18:44:48.884435 18720 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260620 18:44:48.900633 18735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:48.902390 18720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:44:48.902618 18720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-2/wal
uuid: "c8dcaff0cf3c425bab8661e8ba7c3424"
format_stamp: "Formatted at 2026-06-20 18:44:48 on dist-test-slave-k6lg"
server_key: "654463af55929cbeb950555b5d3434ef"
server_key_iv: "847af6b94dde307d6cfb58492e25c4ba"
server_key_version: "kuduclusterkey@0"
I20260620 18:44:48.902858 18720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:48.924420 18720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:48.928218 18720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:48.928532 18720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:48.929469 18720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:44:48.930850 18720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:44:48.930959 18720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:48.931102 18720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:44:48.931172 18720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:48.964496 18720 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.131:32929
I20260620 18:44:48.964550 18847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.131:32929 every 8 connection(s)
I20260620 18:44:48.965827 18720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
I20260620 18:44:48.970189 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 18720
I20260620 18:44:48.970393 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance
I20260620 18:44:48.971024 6022 external_mini_cluster.cc:1428] Setting key 4f6e49857fb8b694937a7f71771e1ec5
Jun 20 18:44:48 dist-test-slave-k6lg krb5kdc[17881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981088, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:44:48.987490 18850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:48.968709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:59979 (local address 127.5.225.190:38707)
0620 18:44:48.969198 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:48.969203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:48.970369 (+ 1166us) server_negotiation.cc:407] Connection header received
0620 18:44:48.972803 (+ 2434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:48.972807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:48.972875 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:48.973013 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:48.974775 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:48.975495 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:48.977092 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:48.977518 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:48.977653 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:48.981093 (+ 3440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:48.981099 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:48.981102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:48.981138 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:48.983925 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:48.984731 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:48.984736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:48.984739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:48.984883 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:48.985448 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:48.985452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:48.985455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:48.985841 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:48.985990 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:48.987043 (+ 1053us) server_negotiation.cc:299] Negotiation successful
0620 18:44:48.987232 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":228,"threads_started":1}
I20260620 18:44:48.988641 18851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:48.969232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:38707 (local address 127.5.225.131:59979)
0620 18:44:48.970074 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:48.970140 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:48.972504 (+ 2364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:48.973300 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:48.973311 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:48.973917 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:48.974612 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:48.974630 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:48.975647 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:48.975652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:48.976938 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:48.976956 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:48.977739 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:48.977748 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:48.977978 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:48.978697 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:48.978719 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:48.980876 (+ 2157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:48.984096 (+ 3220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:48.984106 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:48.984124 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:48.984577 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:48.985004 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:48.985010 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:48.985014 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:48.985305 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:48.985995 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:48.986003 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:48.986562 (+ 559us) client_negotiation.cc:769] Sending connection context
0620 18:44:48.986903 (+ 341us) client_negotiation.cc:240] Negotiation successful
0620 18:44:48.987358 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":290,"threads_started":1}
I20260620 18:44:48.990190 18848 heartbeater.cc:347] Connected to a master server at 127.5.225.190:38707
I20260620 18:44:48.990471 18848 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:48.991166 18848 heartbeater.cc:511] Master 127.5.225.190:38707 requested a full tablet report, sending...
I20260620 18:44:48.992517 18357 ts_manager.cc:194] Registered new tserver with Master: c8dcaff0cf3c425bab8661e8ba7c3424 (127.5.225.131:32929)
I20260620 18:44:48.993310 18357 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.131@KRBTEST.COM'} at 127.5.225.131:59979
I20260620 18:44:48.999228 6022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:44:49.000954 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 18438
Jun 20 18:44:49 dist-test-slave-k6lg krb5kdc[17881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.129@KRBTEST.COM
I20260620 18:44:49.018953 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:49.001773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37344 (local address 127.5.225.129:41987)
0620 18:44:49.002724 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:49.002734 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:49.002768 (+ 34us) server_negotiation.cc:407] Connection header received
0620 18:44:49.002835 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:49.002842 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:49.003047 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:49.003190 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:49.004707 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:49.005903 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:49.007537 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:49.008190 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:49.008299 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:49.012924 (+ 4625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:49.012936 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:49.012950 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:49.012985 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:49.016143 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:49.016906 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:49.016913 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:49.016916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:49.017060 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:49.017434 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:49.017438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:49.017440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:49.017823 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:49.018078 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:49.018390 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 18:44:49.018646 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":788,"thread_start_us":623,"threads_started":1}
I20260620 18:44:49.020198 18566 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:49.106520 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 18438
I20260620 18:44:49.123148 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 18577
Jun 20 18:44:49 dist-test-slave-k6lg krb5kdc[17881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.130@KRBTEST.COM
I20260620 18:44:49.139968 18855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:49.124136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33628 (local address 127.5.225.130:35497)
0620 18:44:49.124682 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:49.124689 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:49.124714 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:44:49.124771 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:49.124777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:49.125006 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:49.125183 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:49.126604 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:49.127616 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:49.128950 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:49.129509 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:49.129652 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:49.134248 (+ 4596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:49.134266 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:49.134281 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:49.134319 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:49.136961 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:49.137670 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:49.137681 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:49.137689 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:49.137830 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:49.138315 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:49.138323 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:49.138337 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:49.138710 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:49.139008 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:49.139364 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 18:44:49.139626 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":284,"threads_started":1}
I20260620 18:44:49.141374 18709 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:49.224298 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 18577
I20260620 18:44:49.239933 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 18720
Jun 20 18:44:49 dist-test-slave-k6lg krb5kdc[17881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.131@KRBTEST.COM
I20260620 18:44:49.256414 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:49.241053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35102 (local address 127.5.225.131:32929)
0620 18:44:49.241773 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:49.241780 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:49.241808 (+ 28us) server_negotiation.cc:407] Connection header received
0620 18:44:49.241866 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:49.241873 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:49.242089 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:49.242328 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:49.243551 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:49.244685 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:49.246102 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:49.246802 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:49.246909 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:49.251165 (+ 4256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:49.251176 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:49.251189 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:49.251220 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:49.253804 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:49.254487 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:49.254494 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:49.254496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:49.254608 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:49.254969 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:49.254973 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:49.254975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:49.255339 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:49.255584 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:49.255925 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 18:44:49.256117 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":399,"threads_started":1}
I20260620 18:44:49.257696 18843 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:49.340713 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 18720
I20260620 18:44:49.356403 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 18325
I20260620 18:44:49.357694 18387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:49.466782 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 18325
I20260620 18:44:49.488312 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:38707
--webserver_interface=127.5.225.190
--webserver_port=34479
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:35267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:38707
--ranger_config_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.225.148:58191/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.225.148:58191/kms with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:49.810266 18860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:49.810563 18860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:49.810750 18860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:49.817996 18860 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:44:49.818161 18860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:49.818298 18860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:49.818403 18860 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:44:49.818495 18860 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:44:49.828910 18860 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:35267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.225.148:58191/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:38707
--ranger_config_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:38707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=34479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:44:49.830331 18860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:49.831674 18860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:49.844555 18865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:49.844590 18866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:49.845144 18868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:49.845551 18860 server_base.cc:1034] running on GCE node
I20260620 18:44:49.846392 18860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:49.847993 18860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:49.849222 18860 hybrid_clock.cc:648] HybridClock initialized: now 1781981089849159 us; error 43 us; skew 500 ppm
I20260620 18:44:49.849596 18860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:49 dist-test-slave-k6lg krb5kdc[17881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981089, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:49.852873 18860 init.cc:377] Logged in from keytab as kudu/127.5.225.190@KRBTEST.COM (short username kudu)
I20260620 18:44:49.854470 18860 webserver.cc:460] Webserver started at http://127.5.225.190:34479/ using document root <none> and password file <none>
I20260620 18:44:49.855167 18860 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:49.855305 18860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 18:44:49 dist-test-slave-k6lg krb5kdc[17881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981089, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for HTTP/127.5.225.148@KRBTEST.COM
I20260620 18:44:49.943532 18860 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260620 18:44:49.946739 18876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:49.948379 18860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:44:49.948583 18860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "e20d976f1e114be4a9476558d845c666"
format_stamp: "Formatted at 2026-06-20 18:44:45 on dist-test-slave-k6lg"
server_key: "6645d28bf18c29bff321460c94665021"
server_key_iv: "096bffbf0e483802719607d1cc9d951f"
server_key_version: "kuduclusterkey@0"
I20260620 18:44:49.949535 18860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:49.967926 18860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:49.971932 18860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:49.972229 18860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:49.990803 18860 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:38707
I20260620 18:44:49.990953 18927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:38707 every 8 connection(s)
I20260620 18:44:49.992290 18860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:44:49.999133 18928 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:44:50.001624 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 18860
I20260620 18:44:50.002363 18928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666: Bootstrap starting.
I20260620 18:44:50.002781 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.225.148:58191/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.129:41987
--local_ip_for_outbound_sockets=127.5.225.129
--tserver_master_addrs=127.5.225.190:38707
--webserver_port=37235
--webserver_interface=127.5.225.129
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:35267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.225.148:58191/kms with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:44:50.021420 18928 log.cc:824] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666: Log is configured to *not* fsync() on all Append() calls
I20260620 18:44:50.031322 18928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:44:50.031901 18928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666: Bootstrap complete.
I20260620 18:44:50.042086 18928 raft_consensus.cc:348] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e20d976f1e114be4a9476558d845c666" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 38707 } }
I20260620 18:44:50.043155 18928 raft_consensus.cc:729] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e20d976f1e114be4a9476558d845c666, State: Initialized, Role: FOLLOWER
I20260620 18:44:50.043906 18928 consensus_queue.cc:260] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e20d976f1e114be4a9476558d845c666" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 38707 } }
I20260620 18:44:50.044167 18928 raft_consensus.cc:388] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:44:50.044309 18928 raft_consensus.cc:482] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:44:50.044508 18928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666 [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:44:50.048764 18928 raft_consensus.cc:504] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e20d976f1e114be4a9476558d845c666" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 38707 } }
I20260620 18:44:50.049276 18928 leader_election.cc:302] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e20d976f1e114be4a9476558d845c666; no voters:
I20260620 18:44:50.049971 18928 leader_election.cc:288] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 18:44:50.050133 18932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 18:44:50.051566 18932 raft_consensus.cc:686] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666 [term 2 LEADER]: Becoming Leader. State: Replica: e20d976f1e114be4a9476558d845c666, State: Running, Role: LEADER
I20260620 18:44:50.052205 18932 consensus_queue.cc:237] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e20d976f1e114be4a9476558d845c666" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 38707 } }
I20260620 18:44:50.052424 18928 sys_catalog.cc:564] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:44:50.055684 18934 sys_catalog.cc:455] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e20d976f1e114be4a9476558d845c666. Latest consensus state: current_term: 2 leader_uuid: "e20d976f1e114be4a9476558d845c666" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e20d976f1e114be4a9476558d845c666" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 38707 } } }
I20260620 18:44:50.056143 18934 sys_catalog.cc:458] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666 [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:50.056674 18933 sys_catalog.cc:455] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e20d976f1e114be4a9476558d845c666" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e20d976f1e114be4a9476558d845c666" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 38707 } } }
I20260620 18:44:50.057004 18933 sys_catalog.cc:458] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666 [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:50.061019 18939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:44:50.067684 18939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:44:50.069836 18939 catalog_manager.cc:1259] Loaded cluster ID: 1944c2fbb2bd48dc8a44fe086b9ea4b1
I20260620 18:44:50.069999 18939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:44:50.074129 18939 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:44:50.075955 18939 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666: Loaded TSK: 0
I20260620 18:44:50.077620 18939 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:44:50.083017 18928 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260620 18:44:50.502735 18930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:50.503242 18930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:50.503451 18930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:50.516753 18930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:50.516979 18930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:50.517220 18930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.129
I20260620 18:44:50.534287 18930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:35267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.225.148:58191/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.129:41987
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.225.129
--webserver_port=37235
--tserver_master_addrs=127.5.225.190:38707
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.129
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:44:50.536226 18930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:50.537982 18930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:50.554636 18967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:50.555795 18966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:50.557154 18930 server_base.cc:1034] running on GCE node
W20260620 18:44:50.557548 18969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:50.558395 18930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:50.559716 18930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:50.560977 18930 hybrid_clock.cc:648] HybridClock initialized: now 1781981090560910 us; error 62 us; skew 500 ppm
I20260620 18:44:50.561340 18930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:50 dist-test-slave-k6lg krb5kdc[17881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981090, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:50.564756 18930 init.cc:377] Logged in from keytab as kudu/127.5.225.129@KRBTEST.COM (short username kudu)
I20260620 18:44:50.566605 18930 webserver.cc:460] Webserver started at http://127.5.225.129:37235/ using document root <none> and password file <none>
I20260620 18:44:50.567361 18930 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:50.567534 18930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 18:44:50 dist-test-slave-k6lg krb5kdc[17881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981090, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for HTTP/127.5.225.148@KRBTEST.COM
I20260620 18:44:50.595996 18930 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.006s sys 0.001s
I20260620 18:44:50.606812 18977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:50.608742 18930 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.007s sys 0.002s
I20260620 18:44:50.608974 18930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-0/wal
uuid: "edc7c06ba14b4d049cc3ac674929e12b"
format_stamp: "Formatted at 2026-06-20 18:44:47 on dist-test-slave-k6lg"
server_key: "59194f8ee02762d0cf263b021bae91cf"
server_key_iv: "a744ba2cc8ad2b85f7028e821d3408c0"
server_key_version: "kuduclusterkey@0"
I20260620 18:44:50.610548 18930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:50.654114 18930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:50.659772 18930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:50.660096 18930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:50.661381 18930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:44:50.663272 18930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:44:50.663384 18930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:50.663491 18930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:44:50.663579 18930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:50.739223 18930 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.129:41987
I20260620 18:44:50.739274 19090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.129:41987 every 8 connection(s)
I20260620 18:44:50.740687 18930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
I20260620 18:44:50.742523 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 18930
I20260620 18:44:50.743428 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.225.148:58191/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.130:35497
--local_ip_for_outbound_sockets=127.5.225.130
--tserver_master_addrs=127.5.225.190:38707
--webserver_port=33301
--webserver_interface=127.5.225.130
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:35267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.225.148:58191/kms with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:44:50 dist-test-slave-k6lg krb5kdc[17881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981090, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:44:50.785598 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:50.744371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:38707 (local address 127.5.225.129:54145)
0620 18:44:50.750497 (+ 6126us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:50.750566 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:50.753851 (+ 3285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:50.756401 (+ 2550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:50.756413 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:50.757084 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:50.757942 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:50.757978 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:50.760514 (+ 2536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:50.760521 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:50.762979 (+ 2458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:50.763007 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:50.764624 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:50.764634 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:50.764981 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:50.765821 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:50.765858 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:50.769106 (+ 3248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:50.780146 (+ 11040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:50.780155 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:50.780174 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:50.780782 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:50.781439 (+ 657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:50.781445 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:50.781449 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:50.781760 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:50.782900 (+ 1140us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:50.782909 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:50.783335 (+ 426us) client_negotiation.cc:769] Sending connection context
0620 18:44:50.783620 (+ 285us) client_negotiation.cc:240] Negotiation successful
0620 18:44:50.784234 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":546,"threads_started":1}
I20260620 18:44:50.787963 19094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:50.745405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:54145 (local address 127.5.225.190:38707)
0620 18:44:50.754727 (+ 9322us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:50.754757 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:50.754807 (+ 50us) server_negotiation.cc:407] Connection header received
0620 18:44:50.755243 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:50.755261 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:50.755776 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:50.756183 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:50.758273 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:50.760316 (+ 2043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:50.763216 (+ 2900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:50.764358 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:50.764579 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:50.772695 (+ 8116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:50.772707 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:50.772733 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:50.772801 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:50.779947 (+ 7146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:50.780978 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:50.780990 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:50.780999 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:50.781241 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:50.781930 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:50.781936 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:50.781940 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:50.782719 (+ 779us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:50.782938 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:50.786474 (+ 3536us) server_negotiation.cc:299] Negotiation successful
0620 18:44:50.786876 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9065,"thread_start_us":8876,"threads_started":1}
I20260620 18:44:50.791685 19091 heartbeater.cc:347] Connected to a master server at 127.5.225.190:38707
I20260620 18:44:50.791980 19091 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:50.792659 19091 heartbeater.cc:511] Master 127.5.225.190:38707 requested a full tablet report, sending...
W20260620 18:44:50.795723 19091 heartbeater.cc:644] Failed to heartbeat to 127.5.225.190:38707 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 18:44:50.795934 19091 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:50.796177 19091 heartbeater.cc:511] Master 127.5.225.190:38707 requested a full tablet report, sending...
I20260620 18:44:50.800725 19091 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:50.801009 19091 heartbeater.cc:511] Master 127.5.225.190:38707 requested a full tablet report, sending...
W20260620 18:44:50.802213 19091 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:44:51.162118 19095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:51.162503 19095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:51.162688 19095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:51.173019 19095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:51.173223 19095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:51.173453 19095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.130
I20260620 18:44:51.186702 19095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:35267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.225.148:58191/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.130:35497
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.225.130
--webserver_port=33301
--tserver_master_addrs=127.5.225.190:38707
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.130
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:44:51.188326 19095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:51.189989 19095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:51.205286 19109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:51.206779 19110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:51.215101 19095 server_base.cc:1034] running on GCE node
W20260620 18:44:51.215571 19112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:51.216398 19095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:51.217674 19095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:51.218900 19095 hybrid_clock.cc:648] HybridClock initialized: now 1781981091218846 us; error 46 us; skew 500 ppm
I20260620 18:44:51.219235 19095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:51 dist-test-slave-k6lg krb5kdc[17881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981091, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:51.222617 19095 init.cc:377] Logged in from keytab as kudu/127.5.225.130@KRBTEST.COM (short username kudu)
I20260620 18:44:51.225124 19095 webserver.cc:460] Webserver started at http://127.5.225.130:33301/ using document root <none> and password file <none>
I20260620 18:44:51.225878 19095 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:51.226038 19095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 18:44:51 dist-test-slave-k6lg krb5kdc[17881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981091, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for HTTP/127.5.225.148@KRBTEST.COM
I20260620 18:44:51.266348 19095 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 18:44:51.271266 19122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:51.273097 19095 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 18:44:51.273283 19095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-1/wal
uuid: "0920e736e9744560bf882d5069c9e91a"
format_stamp: "Formatted at 2026-06-20 18:44:48 on dist-test-slave-k6lg"
server_key: "f1b3e9da0fed3a8e1a98626a4bc697c2"
server_key_iv: "eb6b35c77776ef47f79dfddeffc55ead"
server_key_version: "kuduclusterkey@0"
I20260620 18:44:51.274811 19095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:51.320796 19095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:51.326278 19095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:51.326589 19095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:51.327857 19095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:44:51.329664 19095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:44:51.329802 19095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:51.329911 19095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:44:51.329994 19095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:44:51 dist-test-slave-k6lg krb5kdc[17881](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781981091, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:51.389207 19095 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.130:35497
I20260620 18:44:51.389274 19234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.130:35497 every 8 connection(s)
I20260620 18:44:51.391202 19095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
I20260620 18:44:51.398904 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 19095
I20260620 18:44:51.399686 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.225.148:58191/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.131:32929
--local_ip_for_outbound_sockets=127.5.225.131
--tserver_master_addrs=127.5.225.190:38707
--webserver_port=37855
--webserver_interface=127.5.225.131
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:35267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.225.148:58191/kms with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:44:51 dist-test-slave-k6lg krb5kdc[17881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981091, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:44:51.426537 19238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:51.395957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:58853 (local address 127.5.225.190:38707)
0620 18:44:51.407168 (+ 11211us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:51.407174 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:51.407200 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:44:51.407271 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:51.407276 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:51.407377 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:51.407515 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:51.409579 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:51.410916 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:51.413425 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:51.414098 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:51.414241 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:51.419110 (+ 4869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:51.419118 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:51.419123 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:51.419167 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:51.422893 (+ 3726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:51.423790 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:51.423795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:51.423799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:51.423952 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:51.424457 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:51.424462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:51.424466 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:51.424801 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:51.424928 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:51.425901 (+ 973us) server_negotiation.cc:299] Negotiation successful
0620 18:44:51.426128 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11073,"thread_start_us":10958,"threads_started":1}
I20260620 18:44:51.428689 19237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:51.395273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:38707 (local address 127.5.225.130:58853)
0620 18:44:51.396372 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:51.396450 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:51.400403 (+ 3953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:51.407843 (+ 7440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:51.407857 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:51.408499 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:51.409348 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:51.409376 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:51.411137 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:51.411143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:51.413240 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:51.413257 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:51.414634 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:51.414642 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:51.414966 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:51.415798 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:51.415830 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:51.418772 (+ 2942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:51.423065 (+ 4293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:51.423074 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:51.423093 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:51.423654 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:51.424086 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:51.424091 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:51.424095 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:51.424336 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:51.425023 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:51.425031 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:51.425489 (+ 458us) client_negotiation.cc:769] Sending connection context
0620 18:44:51.426760 (+ 1271us) client_negotiation.cc:240] Negotiation successful
0620 18:44:51.427334 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":707,"threads_started":1}
I20260620 18:44:51.430413 19235 heartbeater.cc:347] Connected to a master server at 127.5.225.190:38707
I20260620 18:44:51.430763 19235 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:51.431582 19235 heartbeater.cc:511] Master 127.5.225.190:38707 requested a full tablet report, sending...
I20260620 18:44:51.433666 18893 ts_manager.cc:194] Registered new tserver with Master: 0920e736e9744560bf882d5069c9e91a (127.5.225.130:35497)
I20260620 18:44:51.435823 18893 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.130@KRBTEST.COM'} at 127.5.225.130:58853
Jun 20 18:44:51 dist-test-slave-k6lg krb5kdc[17881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981091, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for HTTP/127.5.225.148@KRBTEST.COM
I20260620 18:44:51.803706 19091 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:51.804169 19091 heartbeater.cc:511] Master 127.5.225.190:38707 requested a full tablet report, sending...
I20260620 18:44:51.805454 18893 ts_manager.cc:194] Registered new tserver with Master: edc7c06ba14b4d049cc3ac674929e12b (127.5.225.129:41987)
I20260620 18:44:51.806636 18893 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.129@KRBTEST.COM'} at 127.5.225.129:54145
W20260620 18:44:51.869975 19239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:51.870460 19239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:51.870635 19239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:51.881971 19239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:51.882126 19239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:51.882328 19239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.131
I20260620 18:44:51.897524 19239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:35267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.225.148:58191/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.131:32929
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.225.131
--webserver_port=37855
--tserver_master_addrs=127.5.225.190:38707
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.131
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:44:51.899355 19239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:51.901075 19239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:51.918421 19246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:51.922585 19245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:51.925205 19248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:51.925406 19239 server_base.cc:1034] running on GCE node
I20260620 18:44:51.926405 19239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:51.927688 19239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:51.928983 19239 hybrid_clock.cc:648] HybridClock initialized: now 1781981091928919 us; error 61 us; skew 500 ppm
I20260620 18:44:51.929394 19239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:51 dist-test-slave-k6lg krb5kdc[17881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981091, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:51.933614 19239 init.cc:377] Logged in from keytab as kudu/127.5.225.131@KRBTEST.COM (short username kudu)
I20260620 18:44:51.935668 19239 webserver.cc:460] Webserver started at http://127.5.225.131:37855/ using document root <none> and password file <none>
I20260620 18:44:51.936595 19239 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:51.936765 19239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 18:44:51 dist-test-slave-k6lg krb5kdc[17881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981091, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for HTTP/127.5.225.148@KRBTEST.COM
I20260620 18:44:51.968223 19239 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.004s sys 0.008s
I20260620 18:44:51.971987 19256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:51.973775 19239 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 18:44:51.973946 19239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-2/wal
uuid: "c8dcaff0cf3c425bab8661e8ba7c3424"
format_stamp: "Formatted at 2026-06-20 18:44:48 on dist-test-slave-k6lg"
server_key: "654463af55929cbeb950555b5d3434ef"
server_key_iv: "847af6b94dde307d6cfb58492e25c4ba"
server_key_version: "kuduclusterkey@0"
I20260620 18:44:51.975392 19239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:52.000156 19239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:52.005494 19239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:52.005856 19239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:52.007149 19239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:44:52.008962 19239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:44:52.009106 19239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 18:44:52.009238 19239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:44:52.009331 19239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:52.059624 19368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.131:32929 every 8 connection(s)
I20260620 18:44:52.060066 19239 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.131:32929
I20260620 18:44:52.061925 19239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
I20260620 18:44:52.069108 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 19239
Jun 20 18:44:52 dist-test-slave-k6lg krb5kdc[17881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981091, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
Jun 20 18:44:52 dist-test-slave-k6lg krb5kdc[17881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981091, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for HTTP/127.5.225.148@KRBTEST.COM
I20260620 18:44:52.118923 19372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:52.070289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36480 (local address 127.5.225.190:38707)
0620 18:44:52.088795 (+ 18506us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:52.088805 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:52.091931 (+ 3126us) server_negotiation.cc:407] Connection header received
0620 18:44:52.093518 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:52.093523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:52.093611 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:52.095861 (+ 2250us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:52.096060 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:52.097306 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:52.099455 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:52.100102 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:52.104239 (+ 4137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:52.104607 (+ 368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:52.104615 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:52.104623 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:52.104676 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:52.113117 (+ 8441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:52.114385 (+ 1268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:52.114393 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:52.114396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:52.114550 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:52.115082 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:52.115087 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:52.115092 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:52.115521 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:52.118106 (+ 2585us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:52.118287 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 18:44:52.118535 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18306,"thread_start_us":18527,"threads_started":1}
I20260620 18:44:52.132228 19375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:52.089073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:38199 (local address 127.5.225.190:38707)
0620 18:44:52.092496 (+ 3423us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:52.092502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:52.092521 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:44:52.093864 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:52.093870 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:52.093952 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:52.094053 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:52.096329 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:52.097469 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:52.100716 (+ 3247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:52.101452 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:52.101557 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:52.106619 (+ 5062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:52.106627 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:52.106631 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:52.106689 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:52.123840 (+ 17151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:52.125410 (+ 1570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:52.125417 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:52.125420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:52.125572 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:52.129274 (+ 3702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:52.129280 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:52.129283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:52.129603 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:52.129806 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:52.131608 (+ 1802us) server_negotiation.cc:299] Negotiation successful
0620 18:44:52.131861 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3298,"thread_start_us":4046,"threads_started":1}
I20260620 18:44:52.134533 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:52.082826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:38707 (local address 127.5.225.131:38199)
0620 18:44:52.090773 (+ 7947us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:52.090835 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:52.093415 (+ 2580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:52.094542 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:52.094559 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:52.095192 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:52.096110 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:52.096140 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:52.097959 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:52.097964 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:52.100547 (+ 2583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:52.100562 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:52.101950 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:52.101958 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:52.102323 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:52.103379 (+ 1056us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:52.103417 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:52.106287 (+ 2870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:52.124328 (+ 18041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:52.124340 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:52.124362 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:52.125197 (+ 835us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:52.125739 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:52.128812 (+ 3073us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:52.128819 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:52.129092 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:52.129785 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:52.129794 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:52.131067 (+ 1273us) client_negotiation.cc:769] Sending connection context
0620 18:44:52.131445 (+ 378us) client_negotiation.cc:240] Negotiation successful
0620 18:44:52.131964 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7633,"thread_start_us":7392,"threads_started":1}
I20260620 18:44:52.136159 19369 heartbeater.cc:347] Connected to a master server at 127.5.225.190:38707
I20260620 18:44:52.136485 19369 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:52.137177 19369 heartbeater.cc:511] Master 127.5.225.190:38707 requested a full tablet report, sending...
I20260620 18:44:52.138796 18893 ts_manager.cc:194] Registered new tserver with Master: c8dcaff0cf3c425bab8661e8ba7c3424 (127.5.225.131:32929)
I20260620 18:44:52.139637 18893 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.131@KRBTEST.COM'} at 127.5.225.131:38199
I20260620 18:44:52.147286 6022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:44:52.168866 19375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:52.151288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36492 (local address 127.5.225.190:38707)
0620 18:44:52.151470 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:52.151476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:52.152321 (+ 845us) server_negotiation.cc:407] Connection header received
0620 18:44:52.152547 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:52.152552 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:52.152654 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:52.152732 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:52.154932 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:52.155993 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:52.157830 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:52.158512 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:52.158606 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:52.161927 (+ 3321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:52.161940 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:52.161944 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:52.161980 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:52.165507 (+ 3527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:52.166465 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:52.166471 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:52.166475 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:52.166626 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:52.167098 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:52.167104 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:52.167108 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:52.167487 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:52.167699 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:52.168261 (+ 562us) server_negotiation.cc:299] Negotiation successful
0620 18:44:52.168497 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:44:52.175963 18893 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36492:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:44:52.187422 18893 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:44:52.296109 19387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:52.249785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33642 (local address 127.5.225.130:35497)
0620 18:44:52.272697 (+ 22912us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:52.272708 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:52.277323 (+ 4615us) server_negotiation.cc:407] Connection header received
0620 18:44:52.277486 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:52.277494 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:52.277798 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:52.278022 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:52.281674 (+ 3652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:52.283346 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:52.292756 (+ 9410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:52.294533 (+ 1777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:52.294654 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:52.294867 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:52.295615 (+ 748us) server_negotiation.cc:299] Negotiation successful
0620 18:44:52.295797 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22716,"thread_start_us":22942,"threads_started":1}
I20260620 18:44:52.297236 19388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:52.249680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.130:35497 (local address 127.0.0.1:33642)
0620 18:44:52.277170 (+ 27490us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:52.277190 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:52.277327 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:52.280526 (+ 3199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:52.280531 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:52.280601 (+ 70us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:52.280950 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:52.280962 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:52.290344 (+ 9382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:52.290353 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:52.292557 (+ 2204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:52.292572 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:52.295103 (+ 2531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:52.295110 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:52.295328 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:52.295390 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 18:44:52.296646 (+ 1256us) client_negotiation.cc:240] Negotiation successful
0620 18:44:52.296856 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27369,"thread_start_us":27233,"threads_started":1}
I20260620 18:44:52.299751 19385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:52.249096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.131:32929 (local address 127.0.0.1:35104)
0620 18:44:52.275743 (+ 26647us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:52.275817 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:52.276109 (+ 292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:52.290697 (+ 14588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:52.290702 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:52.290733 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:52.291169 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:52.291189 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:52.293142 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:52.293147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:52.295102 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:52.295118 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:52.299031 (+ 3913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:52.299037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:52.299217 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:52.299248 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 18:44:52.299323 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 18:44:52.299448 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26347,"thread_start_us":26203,"threads_started":1}
I20260620 18:44:52.300562 19386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:52.249407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35104 (local address 127.5.225.131:32929)
0620 18:44:52.288647 (+ 39240us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:52.288656 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:52.288690 (+ 34us) server_negotiation.cc:407] Connection header received
0620 18:44:52.288799 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:52.288808 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:52.289088 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:52.289304 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:52.291386 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:52.292980 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:52.295281 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:52.296664 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:52.296780 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:52.296998 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:52.300069 (+ 3071us) server_negotiation.cc:299] Negotiation successful
0620 18:44:52.300240 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39057,"thread_start_us":40131,"threads_started":1}
I20260620 18:44:52.301645 19170 tablet_service.cc:1467] Processing CreateTablet for tablet 372baf43f4aa43d49c4330c6a659cc20 (DEFAULT_TABLE table=test-table [id=e17ae6768b984bbe978915105f060c3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:44:52.303104 19170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 372baf43f4aa43d49c4330c6a659cc20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:52.305429 19304 tablet_service.cc:1467] Processing CreateTablet for tablet 372baf43f4aa43d49c4330c6a659cc20 (DEFAULT_TABLE table=test-table [id=e17ae6768b984bbe978915105f060c3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:44:52.306948 19304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 372baf43f4aa43d49c4330c6a659cc20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:52.314934 19390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:52.251176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37350 (local address 127.5.225.129:41987)
0620 18:44:52.277883 (+ 26707us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:52.277892 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:52.278862 (+ 970us) server_negotiation.cc:407] Connection header received
0620 18:44:52.278979 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:52.278989 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:52.279223 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:52.279405 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:52.286271 (+ 6866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:52.287892 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:52.297104 (+ 9212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:52.298558 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:52.298680 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:52.298878 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:52.314376 (+ 15498us) server_negotiation.cc:299] Negotiation successful
0620 18:44:52.314594 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26497,"thread_start_us":26791,"threads_started":1}
I20260620 18:44:52.301815 19389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:52.250908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:41987 (local address 127.0.0.1:37350)
0620 18:44:52.278615 (+ 27707us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:52.278634 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:52.278742 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:52.279671 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:52.279679 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:52.279708 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:52.280110 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:52.280124 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:52.288106 (+ 7982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:52.288115 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:52.290504 (+ 2389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:52.290535 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:52.301061 (+ 10526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:52.301067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:52.301239 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:52.301273 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 18:44:52.301360 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 18:44:52.301514 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27581,"thread_start_us":27438,"threads_started":1}
I20260620 18:44:52.319404 19026 tablet_service.cc:1467] Processing CreateTablet for tablet 372baf43f4aa43d49c4330c6a659cc20 (DEFAULT_TABLE table=test-table [id=e17ae6768b984bbe978915105f060c3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:44:52.320827 19026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 372baf43f4aa43d49c4330c6a659cc20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:52.331789 19391 tablet_bootstrap.cc:492] T 372baf43f4aa43d49c4330c6a659cc20 P c8dcaff0cf3c425bab8661e8ba7c3424: Bootstrap starting.
I20260620 18:44:52.339541 19391 tablet_bootstrap.cc:654] T 372baf43f4aa43d49c4330c6a659cc20 P c8dcaff0cf3c425bab8661e8ba7c3424: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:52.341048 19391 log.cc:824] T 372baf43f4aa43d49c4330c6a659cc20 P c8dcaff0cf3c425bab8661e8ba7c3424: Log is configured to *not* fsync() on all Append() calls
I20260620 18:44:52.351046 19391 tablet_bootstrap.cc:492] T 372baf43f4aa43d49c4330c6a659cc20 P c8dcaff0cf3c425bab8661e8ba7c3424: No bootstrap required, opened a new log
I20260620 18:44:52.351496 19391 ts_tablet_manager.cc:1397] T 372baf43f4aa43d49c4330c6a659cc20 P c8dcaff0cf3c425bab8661e8ba7c3424: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.003s
I20260620 18:44:52.361572 19393 tablet_bootstrap.cc:492] T 372baf43f4aa43d49c4330c6a659cc20 P 0920e736e9744560bf882d5069c9e91a: Bootstrap starting.
I20260620 18:44:52.364637 19391 raft_consensus.cc:348] T 372baf43f4aa43d49c4330c6a659cc20 P c8dcaff0cf3c425bab8661e8ba7c3424 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8dcaff0cf3c425bab8661e8ba7c3424" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 32929 } } peers { permanent_uuid: "0920e736e9744560bf882d5069c9e91a" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 35497 } } peers { permanent_uuid: "edc7c06ba14b4d049cc3ac674929e12b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 41987 } }
I20260620 18:44:52.365351 19391 raft_consensus.cc:374] T 372baf43f4aa43d49c4330c6a659cc20 P c8dcaff0cf3c425bab8661e8ba7c3424 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:52.365525 19391 raft_consensus.cc:729] T 372baf43f4aa43d49c4330c6a659cc20 P c8dcaff0cf3c425bab8661e8ba7c3424 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8dcaff0cf3c425bab8661e8ba7c3424, State: Initialized, Role: FOLLOWER
I20260620 18:44:52.366431 19391 consensus_queue.cc:260] T 372baf43f4aa43d49c4330c6a659cc20 P c8dcaff0cf3c425bab8661e8ba7c3424 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8dcaff0cf3c425bab8661e8ba7c3424" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 32929 } } peers { permanent_uuid: "0920e736e9744560bf882d5069c9e91a" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 35497 } } peers { permanent_uuid: "edc7c06ba14b4d049cc3ac674929e12b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 41987 } }
I20260620 18:44:52.375326 19394 tablet_bootstrap.cc:492] T 372baf43f4aa43d49c4330c6a659cc20 P edc7c06ba14b4d049cc3ac674929e12b: Bootstrap starting.
I20260620 18:44:52.376250 19391 ts_tablet_manager.cc:1428] T 372baf43f4aa43d49c4330c6a659cc20 P c8dcaff0cf3c425bab8661e8ba7c3424: Time spent starting tablet: real 0.025s user 0.018s sys 0.001s
I20260620 18:44:52.376716 19369 heartbeater.cc:503] Master 127.5.225.190:38707 was elected leader, sending a full tablet report...
I20260620 18:44:52.378755 19393 tablet_bootstrap.cc:654] T 372baf43f4aa43d49c4330c6a659cc20 P 0920e736e9744560bf882d5069c9e91a: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:52.379194 19394 tablet_bootstrap.cc:654] T 372baf43f4aa43d49c4330c6a659cc20 P edc7c06ba14b4d049cc3ac674929e12b: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:52.380424 19394 log.cc:824] T 372baf43f4aa43d49c4330c6a659cc20 P edc7c06ba14b4d049cc3ac674929e12b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:44:52.394117 19394 tablet_bootstrap.cc:492] T 372baf43f4aa43d49c4330c6a659cc20 P edc7c06ba14b4d049cc3ac674929e12b: No bootstrap required, opened a new log
I20260620 18:44:52.394373 19393 log.cc:824] T 372baf43f4aa43d49c4330c6a659cc20 P 0920e736e9744560bf882d5069c9e91a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:44:52.395011 19394 ts_tablet_manager.cc:1397] T 372baf43f4aa43d49c4330c6a659cc20 P edc7c06ba14b4d049cc3ac674929e12b: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.000s
I20260620 18:44:52.405786 19394 raft_consensus.cc:348] T 372baf43f4aa43d49c4330c6a659cc20 P edc7c06ba14b4d049cc3ac674929e12b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8dcaff0cf3c425bab8661e8ba7c3424" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 32929 } } peers { permanent_uuid: "0920e736e9744560bf882d5069c9e91a" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 35497 } } peers { permanent_uuid: "edc7c06ba14b4d049cc3ac674929e12b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 41987 } }
I20260620 18:44:52.406433 19394 raft_consensus.cc:374] T 372baf43f4aa43d49c4330c6a659cc20 P edc7c06ba14b4d049cc3ac674929e12b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:52.406597 19394 raft_consensus.cc:729] T 372baf43f4aa43d49c4330c6a659cc20 P edc7c06ba14b4d049cc3ac674929e12b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edc7c06ba14b4d049cc3ac674929e12b, State: Initialized, Role: FOLLOWER
I20260620 18:44:52.407482 19394 consensus_queue.cc:260] T 372baf43f4aa43d49c4330c6a659cc20 P edc7c06ba14b4d049cc3ac674929e12b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8dcaff0cf3c425bab8661e8ba7c3424" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 32929 } } peers { permanent_uuid: "0920e736e9744560bf882d5069c9e91a" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 35497 } } peers { permanent_uuid: "edc7c06ba14b4d049cc3ac674929e12b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 41987 } }
I20260620 18:44:52.407981 19393 tablet_bootstrap.cc:492] T 372baf43f4aa43d49c4330c6a659cc20 P 0920e736e9744560bf882d5069c9e91a: No bootstrap required, opened a new log
I20260620 18:44:52.413013 19393 ts_tablet_manager.cc:1397] T 372baf43f4aa43d49c4330c6a659cc20 P 0920e736e9744560bf882d5069c9e91a: Time spent bootstrapping tablet: real 0.047s user 0.012s sys 0.005s
I20260620 18:44:52.429917 19091 heartbeater.cc:503] Master 127.5.225.190:38707 was elected leader, sending a full tablet report...
I20260620 18:44:52.430810 19394 ts_tablet_manager.cc:1428] T 372baf43f4aa43d49c4330c6a659cc20 P edc7c06ba14b4d049cc3ac674929e12b: Time spent starting tablet: real 0.036s user 0.022s sys 0.012s
I20260620 18:44:52.430939 19393 raft_consensus.cc:348] T 372baf43f4aa43d49c4330c6a659cc20 P 0920e736e9744560bf882d5069c9e91a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8dcaff0cf3c425bab8661e8ba7c3424" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 32929 } } peers { permanent_uuid: "0920e736e9744560bf882d5069c9e91a" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 35497 } } peers { permanent_uuid: "edc7c06ba14b4d049cc3ac674929e12b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 41987 } }
I20260620 18:44:52.431548 19393 raft_consensus.cc:374] T 372baf43f4aa43d49c4330c6a659cc20 P 0920e736e9744560bf882d5069c9e91a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:52.431715 19393 raft_consensus.cc:729] T 372baf43f4aa43d49c4330c6a659cc20 P 0920e736e9744560bf882d5069c9e91a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0920e736e9744560bf882d5069c9e91a, State: Initialized, Role: FOLLOWER
I20260620 18:44:52.432642 19393 consensus_queue.cc:260] T 372baf43f4aa43d49c4330c6a659cc20 P 0920e736e9744560bf882d5069c9e91a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8dcaff0cf3c425bab8661e8ba7c3424" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 32929 } } peers { permanent_uuid: "0920e736e9744560bf882d5069c9e91a" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 35497 } } peers { permanent_uuid: "edc7c06ba14b4d049cc3ac674929e12b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 41987 } }
I20260620 18:44:52.437289 19235 heartbeater.cc:503] Master 127.5.225.190:38707 was elected leader, sending a full tablet report...
I20260620 18:44:52.438069 19393 ts_tablet_manager.cc:1428] T 372baf43f4aa43d49c4330c6a659cc20 P 0920e736e9744560bf882d5069c9e91a: Time spent starting tablet: real 0.025s user 0.019s sys 0.000s
I20260620 18:44:52.457664 19398 raft_consensus.cc:482] T 372baf43f4aa43d49c4330c6a659cc20 P edc7c06ba14b4d049cc3ac674929e12b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:44:52.458027 19398 raft_consensus.cc:504] T 372baf43f4aa43d49c4330c6a659cc20 P edc7c06ba14b4d049cc3ac674929e12b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8dcaff0cf3c425bab8661e8ba7c3424" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 32929 } } peers { permanent_uuid: "0920e736e9744560bf882d5069c9e91a" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 35497 } } peers { permanent_uuid: "edc7c06ba14b4d049cc3ac674929e12b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 41987 } }
I20260620 18:44:52.469211 19398 leader_election.cc:288] T 372baf43f4aa43d49c4330c6a659cc20 P edc7c06ba14b4d049cc3ac674929e12b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8dcaff0cf3c425bab8661e8ba7c3424 (127.5.225.131:32929), 0920e736e9744560bf882d5069c9e91a (127.5.225.130:35497)
I20260620 18:44:52.479185 19387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:52.461183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:35289 (local address 127.5.225.130:35497)
0620 18:44:52.461674 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:52.461683 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:52.471582 (+ 9899us) server_negotiation.cc:407] Connection header received
0620 18:44:52.471882 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:52.471888 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:52.471976 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:52.472041 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:52.472845 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:52.473894 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:52.476672 (+ 2778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:52.477951 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:52.478079 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:52.478152 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:52.478782 (+ 630us) server_negotiation.cc:299] Negotiation successful
0620 18:44:52.478922 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303}
I20260620 18:44:52.479947 19401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:52.460587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.130:35497 (local address 127.5.225.129:35289)
0620 18:44:52.471445 (+ 10858us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:52.471470 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:52.471712 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:52.472241 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:52.472246 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:52.472274 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:52.472664 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:52.472676 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:52.474093 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:52.474099 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:52.476486 (+ 2387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:52.476502 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:52.478393 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:52.478399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:52.478588 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:52.478626 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 18:44:52.479411 (+ 785us) client_negotiation.cc:240] Negotiation successful
0620 18:44:52.479554 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10690,"thread_start_us":14485,"threads_started":1}
I20260620 18:44:52.481175 19190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372baf43f4aa43d49c4330c6a659cc20" candidate_uuid: "edc7c06ba14b4d049cc3ac674929e12b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0920e736e9744560bf882d5069c9e91a" is_pre_election: true
I20260620 18:44:52.481884 19190 raft_consensus.cc:2454] T 372baf43f4aa43d49c4330c6a659cc20 P 0920e736e9744560bf882d5069c9e91a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edc7c06ba14b4d049cc3ac674929e12b in term 0.
I20260620 18:44:52.482833 18978 leader_election.cc:302] T 372baf43f4aa43d49c4330c6a659cc20 P edc7c06ba14b4d049cc3ac674929e12b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0920e736e9744560bf882d5069c9e91a, edc7c06ba14b4d049cc3ac674929e12b; no voters:
I20260620 18:44:52.483520 19398 raft_consensus.cc:2781] T 372baf43f4aa43d49c4330c6a659cc20 P edc7c06ba14b4d049cc3ac674929e12b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:44:52.483744 19398 raft_consensus.cc:482] T 372baf43f4aa43d49c4330c6a659cc20 P edc7c06ba14b4d049cc3ac674929e12b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:44:52.483891 19398 raft_consensus.cc:3037] T 372baf43f4aa43d49c4330c6a659cc20 P edc7c06ba14b4d049cc3ac674929e12b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:52.484822 19400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:52.460652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.131:32929 (local address 127.5.225.129:34295)
0620 18:44:52.474937 (+ 14285us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:52.474965 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:52.475345 (+ 380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:52.475878 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:52.475882 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:52.475921 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:52.476338 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:52.476350 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:52.477828 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:52.477834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:52.481988 (+ 4154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:52.482004 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:52.483925 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:52.483932 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:52.484122 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:52.484156 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 18:44:52.484255 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 18:44:52.484410 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14135,"thread_start_us":10142,"threads_started":1}
I20260620 18:44:52.485494 19386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:52.461122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:34295 (local address 127.5.225.131:32929)
0620 18:44:52.461451 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:52.461461 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:52.475066 (+ 13605us) server_negotiation.cc:407] Connection header received
0620 18:44:52.475527 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:52.475532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:52.475613 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:52.475682 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:52.476546 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:52.477636 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:52.482284 (+ 4648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:52.483509 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:52.483676 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:52.483751 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:52.485102 (+ 1351us) server_negotiation.cc:299] Negotiation successful
0620 18:44:52.485247 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260620 18:44:52.486476 19324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372baf43f4aa43d49c4330c6a659cc20" candidate_uuid: "edc7c06ba14b4d049cc3ac674929e12b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8dcaff0cf3c425bab8661e8ba7c3424" is_pre_election: true
I20260620 18:44:52.487038 19324 raft_consensus.cc:2454] T 372baf43f4aa43d49c4330c6a659cc20 P c8dcaff0cf3c425bab8661e8ba7c3424 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edc7c06ba14b4d049cc3ac674929e12b in term 0.
I20260620 18:44:52.488272 19398 raft_consensus.cc:504] T 372baf43f4aa43d49c4330c6a659cc20 P edc7c06ba14b4d049cc3ac674929e12b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8dcaff0cf3c425bab8661e8ba7c3424" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 32929 } } peers { permanent_uuid: "0920e736e9744560bf882d5069c9e91a" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 35497 } } peers { permanent_uuid: "edc7c06ba14b4d049cc3ac674929e12b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 41987 } }
I20260620 18:44:52.489029 19398 leader_election.cc:288] T 372baf43f4aa43d49c4330c6a659cc20 P edc7c06ba14b4d049cc3ac674929e12b [CANDIDATE]: Term 1 election: Requested vote from peers c8dcaff0cf3c425bab8661e8ba7c3424 (127.5.225.131:32929), 0920e736e9744560bf882d5069c9e91a (127.5.225.130:35497)
I20260620 18:44:52.489439 19324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372baf43f4aa43d49c4330c6a659cc20" candidate_uuid: "edc7c06ba14b4d049cc3ac674929e12b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8dcaff0cf3c425bab8661e8ba7c3424"
I20260620 18:44:52.489630 19324 raft_consensus.cc:3037] T 372baf43f4aa43d49c4330c6a659cc20 P c8dcaff0cf3c425bab8661e8ba7c3424 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:52.489843 19190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372baf43f4aa43d49c4330c6a659cc20" candidate_uuid: "edc7c06ba14b4d049cc3ac674929e12b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0920e736e9744560bf882d5069c9e91a"
I20260620 18:44:52.490049 19190 raft_consensus.cc:3037] T 372baf43f4aa43d49c4330c6a659cc20 P 0920e736e9744560bf882d5069c9e91a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:52.492590 19324 raft_consensus.cc:2454] T 372baf43f4aa43d49c4330c6a659cc20 P c8dcaff0cf3c425bab8661e8ba7c3424 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edc7c06ba14b4d049cc3ac674929e12b in term 1.
I20260620 18:44:52.493109 18980 leader_election.cc:302] T 372baf43f4aa43d49c4330c6a659cc20 P edc7c06ba14b4d049cc3ac674929e12b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8dcaff0cf3c425bab8661e8ba7c3424, edc7c06ba14b4d049cc3ac674929e12b; no voters:
I20260620 18:44:52.493175 19190 raft_consensus.cc:2454] T 372baf43f4aa43d49c4330c6a659cc20 P 0920e736e9744560bf882d5069c9e91a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edc7c06ba14b4d049cc3ac674929e12b in term 1.
I20260620 18:44:52.493563 19398 raft_consensus.cc:2781] T 372baf43f4aa43d49c4330c6a659cc20 P edc7c06ba14b4d049cc3ac674929e12b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:44:52.494222 19398 raft_consensus.cc:686] T 372baf43f4aa43d49c4330c6a659cc20 P edc7c06ba14b4d049cc3ac674929e12b [term 1 LEADER]: Becoming Leader. State: Replica: edc7c06ba14b4d049cc3ac674929e12b, State: Running, Role: LEADER
W20260620 18:44:52.494768 19092 tablet.cc:2369] T 372baf43f4aa43d49c4330c6a659cc20 P edc7c06ba14b4d049cc3ac674929e12b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:44:52.495455 19398 consensus_queue.cc:237] T 372baf43f4aa43d49c4330c6a659cc20 P edc7c06ba14b4d049cc3ac674929e12b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8dcaff0cf3c425bab8661e8ba7c3424" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 32929 } } peers { permanent_uuid: "0920e736e9744560bf882d5069c9e91a" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 35497 } } peers { permanent_uuid: "edc7c06ba14b4d049cc3ac674929e12b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 41987 } }
I20260620 18:44:52.501492 18893 catalog_manager.cc:5495] T 372baf43f4aa43d49c4330c6a659cc20 P edc7c06ba14b4d049cc3ac674929e12b reported cstate change: term changed from 0 to 1, leader changed from <none> to edc7c06ba14b4d049cc3ac674929e12b (127.5.225.129). New cstate: current_term: 1 leader_uuid: "edc7c06ba14b4d049cc3ac674929e12b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8dcaff0cf3c425bab8661e8ba7c3424" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 32929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0920e736e9744560bf882d5069c9e91a" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 35497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "edc7c06ba14b4d049cc3ac674929e12b" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 41987 } health_report { overall_health: HEALTHY } } }
I20260620 18:44:52.523181 19390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:52.518065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37358 (local address 127.5.225.129:41987)
0620 18:44:52.518291 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:52.518295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:52.518343 (+ 48us) server_negotiation.cc:407] Connection header received
0620 18:44:52.518539 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:52.518543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:52.518629 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:52.518750 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:44:52.519074 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:52.519790 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:52.521074 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:52.521478 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:52.521570 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:52.522002 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:44:52.522566 (+ 564us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:44:52.522681 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:52.522799 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 18:44:52.522863 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:44:52.531255 19190 raft_consensus.cc:1261] T 372baf43f4aa43d49c4330c6a659cc20 P 0920e736e9744560bf882d5069c9e91a [term 1 FOLLOWER]: Refusing update from remote peer edc7c06ba14b4d049cc3ac674929e12b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:44:52.531255 19324 raft_consensus.cc:1261] T 372baf43f4aa43d49c4330c6a659cc20 P c8dcaff0cf3c425bab8661e8ba7c3424 [term 1 FOLLOWER]: Refusing update from remote peer edc7c06ba14b4d049cc3ac674929e12b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:44:52.532305 19406 consensus_queue.cc:1035] T 372baf43f4aa43d49c4330c6a659cc20 P edc7c06ba14b4d049cc3ac674929e12b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8dcaff0cf3c425bab8661e8ba7c3424" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 32929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:44:52.532644 19398 consensus_queue.cc:1035] T 372baf43f4aa43d49c4330c6a659cc20 P edc7c06ba14b4d049cc3ac674929e12b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0920e736e9744560bf882d5069c9e91a" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 35497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:44:52.561681 18892 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:44:52.562064 18892 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:44:52.566219 18892 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e20d976f1e114be4a9476558d845c666: Sending DeleteTablet for 3 replicas of tablet 372baf43f4aa43d49c4330c6a659cc20
I20260620 18:44:52.567492 19026 tablet_service.cc:1514] Processing DeleteTablet for tablet 372baf43f4aa43d49c4330c6a659cc20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:44:52 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:37350
I20260620 18:44:52.567479 19170 tablet_service.cc:1514] Processing DeleteTablet for tablet 372baf43f4aa43d49c4330c6a659cc20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:44:52 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:33642
I20260620 18:44:52.567502 19304 tablet_service.cc:1514] Processing DeleteTablet for tablet 372baf43f4aa43d49c4330c6a659cc20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:44:52 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:35104
I20260620 18:44:52.577889 19422 tablet_replica.cc:330] T 372baf43f4aa43d49c4330c6a659cc20 P edc7c06ba14b4d049cc3ac674929e12b: stopping tablet replica
I20260620 18:44:52.579875 19422 raft_consensus.cc:2229] T 372baf43f4aa43d49c4330c6a659cc20 P edc7c06ba14b4d049cc3ac674929e12b [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:44:52.580087 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 18930
I20260620 18:44:52.581135 19422 raft_consensus.cc:2258] T 372baf43f4aa43d49c4330c6a659cc20 P edc7c06ba14b4d049cc3ac674929e12b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:44:52.584851 19422 ts_tablet_manager.cc:1905] T 372baf43f4aa43d49c4330c6a659cc20 P edc7c06ba14b4d049cc3ac674929e12b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:44:52.591933 19423 tablet_replica.cc:330] T 372baf43f4aa43d49c4330c6a659cc20 P 0920e736e9744560bf882d5069c9e91a: stopping tablet replica
I20260620 18:44:52.592501 19424 tablet_replica.cc:330] T 372baf43f4aa43d49c4330c6a659cc20 P c8dcaff0cf3c425bab8661e8ba7c3424: stopping tablet replica
I20260620 18:44:52.592909 19424 raft_consensus.cc:2229] T 372baf43f4aa43d49c4330c6a659cc20 P c8dcaff0cf3c425bab8661e8ba7c3424 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:44:52.593431 19424 raft_consensus.cc:2258] T 372baf43f4aa43d49c4330c6a659cc20 P c8dcaff0cf3c425bab8661e8ba7c3424 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:44:52.594120 19423 raft_consensus.cc:2229] T 372baf43f4aa43d49c4330c6a659cc20 P 0920e736e9744560bf882d5069c9e91a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:44:52.594587 19423 raft_consensus.cc:2258] T 372baf43f4aa43d49c4330c6a659cc20 P 0920e736e9744560bf882d5069c9e91a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:44:52.595691 19424 ts_tablet_manager.cc:1905] T 372baf43f4aa43d49c4330c6a659cc20 P c8dcaff0cf3c425bab8661e8ba7c3424: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:44:52.598062 19422 ts_tablet_manager.cc:1918] T 372baf43f4aa43d49c4330c6a659cc20 P edc7c06ba14b4d049cc3ac674929e12b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:44:52.598260 19422 log.cc:1196] T 372baf43f4aa43d49c4330c6a659cc20 P edc7c06ba14b4d049cc3ac674929e12b: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-0/wal/wals/372baf43f4aa43d49c4330c6a659cc20
I20260620 18:44:52.598762 19422 ts_tablet_manager.cc:1939] T 372baf43f4aa43d49c4330c6a659cc20 P edc7c06ba14b4d049cc3ac674929e12b: Deleting consensus metadata
I20260620 18:44:52.600865 18878 catalog_manager.cc:4841] TS edc7c06ba14b4d049cc3ac674929e12b (127.5.225.129:41987): tablet 372baf43f4aa43d49c4330c6a659cc20 (table test-table [id=e17ae6768b984bbe978915105f060c3d]) successfully deleted
I20260620 18:44:52.602371 19423 ts_tablet_manager.cc:1905] T 372baf43f4aa43d49c4330c6a659cc20 P 0920e736e9744560bf882d5069c9e91a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:44:52.602686 19424 ts_tablet_manager.cc:1918] T 372baf43f4aa43d49c4330c6a659cc20 P c8dcaff0cf3c425bab8661e8ba7c3424: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:44:52.602878 19424 log.cc:1196] T 372baf43f4aa43d49c4330c6a659cc20 P c8dcaff0cf3c425bab8661e8ba7c3424: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-2/wal/wals/372baf43f4aa43d49c4330c6a659cc20
I20260620 18:44:52.603423 19424 ts_tablet_manager.cc:1939] T 372baf43f4aa43d49c4330c6a659cc20 P c8dcaff0cf3c425bab8661e8ba7c3424: Deleting consensus metadata
I20260620 18:44:52.605225 18879 catalog_manager.cc:4841] TS c8dcaff0cf3c425bab8661e8ba7c3424 (127.5.225.131:32929): tablet 372baf43f4aa43d49c4330c6a659cc20 (table test-table [id=e17ae6768b984bbe978915105f060c3d]) successfully deleted
I20260620 18:44:52.609860 19423 ts_tablet_manager.cc:1918] T 372baf43f4aa43d49c4330c6a659cc20 P 0920e736e9744560bf882d5069c9e91a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:44:52.610044 19423 log.cc:1196] T 372baf43f4aa43d49c4330c6a659cc20 P 0920e736e9744560bf882d5069c9e91a: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980894474392-6022-0/minicluster-data/ts-1/wal/wals/372baf43f4aa43d49c4330c6a659cc20
I20260620 18:44:52.610599 19423 ts_tablet_manager.cc:1939] T 372baf43f4aa43d49c4330c6a659cc20 P 0920e736e9744560bf882d5069c9e91a: Deleting consensus metadata
I20260620 18:44:52.612164 19390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:52.583931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37370 (local address 127.5.225.129:41987)
0620 18:44:52.584123 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:52.584133 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:52.584154 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:44:52.584216 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:52.584221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:52.584309 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:52.584442 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:52.586660 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:52.587876 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:52.589825 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:52.590637 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:52.590793 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:52.602559 (+ 11766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:52.602572 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:52.602590 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:52.602642 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:52.608350 (+ 5708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:52.609409 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:52.609418 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:52.609423 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:52.609619 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:52.610117 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:52.610122 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:52.610125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:52.610639 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:52.610943 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:52.611536 (+ 593us) server_negotiation.cc:299] Negotiation successful
0620 18:44:52.611787 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:44:52.612320 18877 catalog_manager.cc:4841] TS 0920e736e9744560bf882d5069c9e91a (127.5.225.130:35497): tablet 372baf43f4aa43d49c4330c6a659cc20 (table test-table [id=e17ae6768b984bbe978915105f060c3d]) successfully deleted
I20260620 18:44:52.612943 19086 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:52.711273 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 18930
I20260620 18:44:52.729926 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 19095
I20260620 18:44:52.747000 19387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:52.730708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33654 (local address 127.5.225.130:35497)
0620 18:44:52.730884 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:52.730888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:52.730905 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:44:52.731064 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:52.731068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:52.731129 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:52.731212 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:52.732477 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:52.733197 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:52.734487 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:52.734864 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:52.734956 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:52.741515 (+ 6559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:52.741529 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:52.741545 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:52.741585 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:52.744393 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:52.745014 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:52.745021 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:52.745023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:52.745163 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:52.745516 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:52.745520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:52.745522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:52.745867 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:52.746078 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:52.746448 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 18:44:52.746669 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:44:52.747620 19230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:52.838469 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 19095
I20260620 18:44:52.855878 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 19239
I20260620 18:44:52.869207 19386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:52.856750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35116 (local address 127.5.225.131:32929)
0620 18:44:52.856937 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:52.856942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:52.857027 (+ 85us) server_negotiation.cc:407] Connection header received
0620 18:44:52.857173 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:52.857177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:52.857243 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:52.857343 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:52.858727 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:52.859555 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:52.860898 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:52.861310 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:52.861406 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:52.864071 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:52.864087 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:52.864103 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:52.864139 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:52.866881 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:52.867512 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:52.867520 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:52.867522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:52.867647 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:52.867988 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:52.867992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:52.867994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:52.868337 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:52.868579 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:52.868801 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 18:44:52.868963 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:44:52.869874 19364 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:52.958635 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 19239
I20260620 18:44:52.975363 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 18860
I20260620 18:44:52.976665 18923 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:53.093094 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 18860
I20260620 18:44:53.117810 6022 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 18:44:53.452770 6022 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 18:44:53.453049 6022 mini_ranger.cc:66] Stopping Ranger...
I20260620 18:44:53.798370 6022 mini_ranger.cc:68] Stopped Ranger
2026-06-20 18:44:53.798 UTC [17911] LOG: received smart shutdown request
2026-06-20 18:44:53.801 UTC [17911] LOG: background worker "logical replication launcher" (PID 17920) exited with exit code 1
2026-06-20 18:44:53.801 UTC [17915] LOG: shutting down
2026-06-20 18:44:53.967 UTC [17911] LOG: database system is shut down
2026-06-20T18:44:53Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52690 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:44:54 dist-test-slave-k6lg krb5kdc[19437](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:44:54 dist-test-slave-k6lg krb5kdc[19437](info): set up 2 sockets
Jun 20 18:44:54 dist-test-slave-k6lg krb5kdc[19437](info): commencing operation
krb5kdc: starting...
W20260620 18:44:55.218603 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:44:55 dist-test-slave-k6lg krb5kdc[19437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:44:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:44:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
I20260620 18:44:55.452085 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:46693
--webserver_interface=127.5.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:34127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:46693,127.5.225.189:36121,127.5.225.188:40579
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:55.777915 19450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:55.778324 19450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:55.778488 19450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:55.785831 19450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:44:55.785995 19450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:55.786109 19450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:55.786248 19450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:44:55.786334 19450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:44:55.796435 19450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:34127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:46693,127.5.225.189:36121,127.5.225.188:40579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:46693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:44:55.797833 19450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:55.799217 19450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:55.811693 19455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:55.811671 19458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:55.811618 19456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:55.812485 19450 server_base.cc:1034] running on GCE node
I20260620 18:44:55.813297 19450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:55.815028 19450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:55.816323 19450 hybrid_clock.cc:648] HybridClock initialized: now 1781981095816220 us; error 71 us; skew 500 ppm
I20260620 18:44:55.816758 19450 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:55 dist-test-slave-k6lg krb5kdc[19437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981095, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:55.820331 19450 init.cc:377] Logged in from keytab as kudu/127.5.225.190@KRBTEST.COM (short username kudu)
I20260620 18:44:55.821974 19450 webserver.cc:460] Webserver started at http://127.5.225.190:45201/ using document root <none> and password file <none>
I20260620 18:44:55.822777 19450 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:55.822928 19450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:55.823246 19450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:55.826514 19450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "4b40cedf94f8499fa151f6501d7f5083"
format_stamp: "Formatted at 2026-06-20 18:44:55 on dist-test-slave-k6lg"
server_key: "edb128e34f6954d6dbf87bf566a063dd"
server_key_iv: "719fcaed4a08a852d690d8db01effc10"
server_key_version: "encryptionkey@0"
I20260620 18:44:55.827380 19450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "4b40cedf94f8499fa151f6501d7f5083"
format_stamp: "Formatted at 2026-06-20 18:44:55 on dist-test-slave-k6lg"
server_key: "edb128e34f6954d6dbf87bf566a063dd"
server_key_iv: "719fcaed4a08a852d690d8db01effc10"
server_key_version: "encryptionkey@0"
I20260620 18:44:55.832937 19450 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:44:55.836611 19465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:55.838088 19450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 18:44:55.838339 19450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "4b40cedf94f8499fa151f6501d7f5083"
format_stamp: "Formatted at 2026-06-20 18:44:55 on dist-test-slave-k6lg"
server_key: "edb128e34f6954d6dbf87bf566a063dd"
server_key_iv: "719fcaed4a08a852d690d8db01effc10"
server_key_version: "encryptionkey@0"
I20260620 18:44:55.838558 19450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:55.851903 19450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:55.856633 19450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:55.856956 19450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:55.875972 19450 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:46693
I20260620 18:44:55.875996 19516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:46693 every 8 connection(s)
I20260620 18:44:55.877434 19450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:44:55.881253 19517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:55.884132 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 19450
I20260620 18:44:55.884384 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:44:55.885083 6022 external_mini_cluster.cc:1428] Setting key c79b02c965437efcf1d251df4c8a49f7
I20260620 18:44:55.888159 19517 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.225.190" port: 46693 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:44:55 dist-test-slave-k6lg krb5kdc[19437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981095, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.225.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.189@KRBTEST.COM" created.
I20260620 18:44:55.911657 19521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:55.891638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59148 (local address 127.5.225.190:46693)
0620 18:44:55.893384 (+ 1746us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:55.893397 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:55.893446 (+ 49us) server_negotiation.cc:407] Connection header received
0620 18:44:55.895105 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:55.895111 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:55.895339 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:55.895553 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:55.896834 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.898032 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.899475 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.900019 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.900220 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:55.903950 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:55.903965 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:55.903984 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:55.904031 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:55.907083 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:55.907890 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:55.907897 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:55.907899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:55.908038 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:55.908429 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:55.908436 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:55.908444 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:55.908934 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:55.909146 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:55.909661 (+ 515us) server_negotiation.cc:299] Negotiation successful
0620 18:44:55.910125 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":1524,"thread_start_us":1312,"threads_started":1}
I20260620 18:44:55.911677 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:55.891637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:46693 (local address 127.0.0.1:59148)
0620 18:44:55.892552 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:55.892605 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:55.894687 (+ 2082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:55.895564 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:55.895571 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:55.896295 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:55.896625 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:55.896642 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.898210 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.898215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:55.899319 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:55.899332 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.900264 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.900271 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:55.900443 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:55.901136 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:55.901158 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:55.903675 (+ 2517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:55.907260 (+ 3585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:55.907274 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:55.907287 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:55.907706 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:55.908150 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:55.908156 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:55.908162 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:55.908311 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:55.909065 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:55.909073 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:55.909286 (+ 213us) client_negotiation.cc:769] Sending connection context
0620 18:44:55.909531 (+ 245us) client_negotiation.cc:240] Negotiation successful
0620 18:44:55.910271 (+ 740us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":230,"thread_start_us":409,"threads_started":1}
I20260620 18:44:55.914601 19517 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.225.189" port: 36121 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:44:55.915721 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:55.915218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:36121 (local address 127.0.0.1:59602)
0620 18:44:55.915427 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:55.915561 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.189:36121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 18:44:55.915982 19468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.225.189:36121: connect: Connection refused (error 111)
I20260620 18:44:55.917706 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:55.917246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:36121 (local address 127.0.0.1:59606)
0620 18:44:55.917441 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:55.917532 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.189:36121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 18:44:55.917906 19517 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.225.189:36121: Network error: Client connection negotiation failed: client connection to 127.5.225.189:36121: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.189.keytab.
Entry for principal kudu/127.5.225.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.189.keytab.
I20260620 18:44:55.934990 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.189:36121
--webserver_interface=127.5.225.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.189.keytab
--principal=kudu/127.5.225.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:34127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:46693,127.5.225.189:36121,127.5.225.188:40579
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:44:55.967258 19517 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.225.189" port: 36121 } attempt: 1
I20260620 18:44:55.968561 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:55.968002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:36121 (local address 127.0.0.1:59610)
0620 18:44:55.968250 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:55.968388 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.189:36121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 18:44:55.969529 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:55.969173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:36121 (local address 127.0.0.1:59622)
0620 18:44:55.969338 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:55.969401 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.189:36121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 18:44:55.969786 19517 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.225.189:36121: Network error: Client connection negotiation failed: client connection to 127.5.225.189:36121: connect: Connection refused (error 111)
I20260620 18:44:56.038071 19517 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.225.189" port: 36121 } attempt: 2
I20260620 18:44:56.039386 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.038861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:36121 (local address 127.0.0.1:59638)
0620 18:44:56.039153 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:56.039242 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.189:36121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260620 18:44:56.040508 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.040142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:36121 (local address 127.0.0.1:59640)
0620 18:44:56.040302 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:56.040381 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.189:36121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 18:44:56.040719 19517 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.225.189:36121: Network error: Client connection negotiation failed: client connection to 127.5.225.189:36121: connect: Connection refused (error 111)
I20260620 18:44:56.132004 19517 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.225.189" port: 36121 } attempt: 3
I20260620 18:44:56.133209 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.132764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:36121 (local address 127.0.0.1:59656)
0620 18:44:56.132973 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:56.133080 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.189:36121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 18:44:56.134344 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.133907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:36121 (local address 127.0.0.1:59664)
0620 18:44:56.134058 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:56.134137 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.189:36121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 18:44:56.134529 19517 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.225.189:36121: Network error: Client connection negotiation failed: client connection to 127.5.225.189:36121: connect: Connection refused (error 111)
W20260620 18:44:56.260538 19524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:56.260896 19524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:56.261061 19524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:56.268813 19524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:44:56.268956 19524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:56.269053 19524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:56.269153 19524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:44:56.269232 19524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:44:56.277782 19517 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.225.189" port: 36121 } attempt: 4
I20260620 18:44:56.279006 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.278468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:36121 (local address 127.0.0.1:59666)
0620 18:44:56.278761 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:56.278863 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.189:36121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260620 18:44:56.279505 19524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:34127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:46693,127.5.225.189:36121,127.5.225.188:40579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.189.keytab
--principal=kudu/127.5.225.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.189:36121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.225.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:44:56.280192 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.279687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:36121 (local address 127.0.0.1:59682)
0620 18:44:56.279939 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:56.280046 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.189:36121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
W20260620 18:44:56.280431 19517 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.225.189:36121: Network error: Client connection negotiation failed: client connection to 127.5.225.189:36121: connect: Connection refused (error 111)
I20260620 18:44:56.281095 19524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:56.282461 19524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:56.294083 19532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:56.294181 19529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:56.294309 19530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:56.295090 19524 server_base.cc:1034] running on GCE node
I20260620 18:44:56.295876 19524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:56.297233 19524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:56.298463 19524 hybrid_clock.cc:648] HybridClock initialized: now 1781981096298400 us; error 60 us; skew 500 ppm
I20260620 18:44:56.298821 19524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:56 dist-test-slave-k6lg krb5kdc[19437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981096, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:56.301889 19524 init.cc:377] Logged in from keytab as kudu/127.5.225.189@KRBTEST.COM (short username kudu)
I20260620 18:44:56.303481 19524 webserver.cc:460] Webserver started at http://127.5.225.189:41463/ using document root <none> and password file <none>
I20260620 18:44:56.304143 19524 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:56.304242 19524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:56.304472 19524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:56.307394 19524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-1/data/instance:
uuid: "aa4fc73911854935bb5eaa622c93914a"
format_stamp: "Formatted at 2026-06-20 18:44:56 on dist-test-slave-k6lg"
server_key: "1b83b115afb7908f67f876e51f52529d"
server_key_iv: "492f5d9fe4685d67d0639de5e5325d86"
server_key_version: "encryptionkey@0"
I20260620 18:44:56.308151 19524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-1/wal/instance:
uuid: "aa4fc73911854935bb5eaa622c93914a"
format_stamp: "Formatted at 2026-06-20 18:44:56 on dist-test-slave-k6lg"
server_key: "1b83b115afb7908f67f876e51f52529d"
server_key_iv: "492f5d9fe4685d67d0639de5e5325d86"
server_key_version: "encryptionkey@0"
I20260620 18:44:56.313107 19524 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 18:44:56.316591 19539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:56.318005 19524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 18:44:56.318145 19524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-1/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-1/wal
uuid: "aa4fc73911854935bb5eaa622c93914a"
format_stamp: "Formatted at 2026-06-20 18:44:56 on dist-test-slave-k6lg"
server_key: "1b83b115afb7908f67f876e51f52529d"
server_key_iv: "492f5d9fe4685d67d0639de5e5325d86"
server_key_version: "encryptionkey@0"
I20260620 18:44:56.318365 19524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:56.348235 19524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:56.352542 19524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:56.352834 19524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:56.371788 19524 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.189:36121
I20260620 18:44:56.371924 19590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.189:36121 every 8 connection(s)
I20260620 18:44:56.373170 19524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-1/data/info.pb
I20260620 18:44:56.376828 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 19524
I20260620 18:44:56.377055 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-1/wal/instance
I20260620 18:44:56.377430 19591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:56.377681 6022 external_mini_cluster.cc:1428] Setting key 31a99b3f859dbaa54dd25ccf357878b7
I20260620 18:44:56.387950 19591 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.225.190" port: 46693 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.225.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.188@KRBTEST.COM" created.
Jun 20 18:44:56 dist-test-slave-k6lg krb5kdc[19437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981096, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.189@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:44:56.410984 19521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.390776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59154 (local address 127.5.225.190:46693)
0620 18:44:56.391023 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:56.391031 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:56.392374 (+ 1343us) server_negotiation.cc:407] Connection header received
0620 18:44:56.394440 (+ 2066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:56.394450 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:56.394548 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:56.394672 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:56.396224 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.397427 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.399257 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.399807 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.399912 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:56.403886 (+ 3974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:56.403896 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:56.403903 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:56.403952 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:56.407807 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:56.408569 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:56.408578 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:56.408584 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:56.408740 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:56.409187 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:56.409191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:56.409193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:56.409482 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:56.409592 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:56.410588 (+ 996us) server_negotiation.cc:299] Negotiation successful
0620 18:44:56.410757 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 18:44:56.412458 19594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.391299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:46693 (local address 127.0.0.1:59154)
0620 18:44:56.392160 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:56.392218 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:56.394090 (+ 1872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:56.394890 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:56.394898 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:56.395440 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:56.396031 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:56.396049 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.397603 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.397608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:56.399091 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:56.399105 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.400029 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.400035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:56.400265 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:56.401014 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:56.401038 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:56.403645 (+ 2607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:56.407966 (+ 4321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:56.407972 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:56.407985 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:56.408430 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:56.408861 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:56.408864 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:56.408867 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:56.409027 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:56.409609 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:56.409618 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:56.410122 (+ 504us) client_negotiation.cc:769] Sending connection context
0620 18:44:56.410486 (+ 364us) client_negotiation.cc:240] Negotiation successful
0620 18:44:56.411035 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":369,"threads_started":1}
I20260620 18:44:56.414615 19591 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.225.189" port: 36121 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:44:56 dist-test-slave-k6lg krb5kdc[19437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981096, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.189@KRBTEST.COM for kudu/127.5.225.189@KRBTEST.COM
I20260620 18:44:56.430998 19594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.415316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:36121 (local address 127.0.0.1:59690)
0620 18:44:56.415655 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:56.415677 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:56.415793 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:56.416747 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:56.416755 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:56.417317 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:56.417544 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:56.417559 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.419008 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.419017 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:56.420793 (+ 1776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:56.420813 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.421705 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.421714 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:56.421989 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:56.422686 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:56.422716 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:56.425123 (+ 2407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:56.428478 (+ 3355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:56.428486 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:56.428490 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:56.428904 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:56.429278 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:56.429282 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:56.429286 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:56.429424 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:56.430146 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:56.430149 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:56.430342 (+ 193us) client_negotiation.cc:769] Sending connection context
0620 18:44:56.430468 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 18:44:56.430662 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20260620 18:44:56.431159 19596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.415567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59690 (local address 127.5.225.189:36121)
0620 18:44:56.416305 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:56.416310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:56.416341 (+ 31us) server_negotiation.cc:407] Connection header received
0620 18:44:56.416385 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:56.416390 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:56.416566 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:56.416687 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:56.417714 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.418848 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.420950 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.421550 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.421646 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:56.425331 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:56.425342 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:56.425354 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:56.425382 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:56.428356 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:56.429028 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:56.429043 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:56.429045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:56.429148 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:56.429589 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:56.429594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:56.429599 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:56.429982 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:56.430305 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:56.430599 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 18:44:56.430857 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":438,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.188.keytab.
Entry for principal kudu/127.5.225.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.188.keytab.
I20260620 18:44:56.433486 19591 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40579 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:44:56.433861 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.188:40579
--webserver_interface=127.5.225.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.188.keytab
--principal=kudu/127.5.225.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:34127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:46693,127.5.225.189:36121,127.5.225.188:40579
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:44:56.434734 19594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.434294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:40579 (local address 127.0.0.1:48518)
0620 18:44:56.434476 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:56.434572 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 18:44:56.435005 19543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40579: connect: Connection refused (error 111)
I20260620 18:44:56.436553 19594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.436229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:40579 (local address 127.0.0.1:35724)
0620 18:44:56.436371 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:56.436457 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 18:44:56.436781 19591 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.225.188:40579: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40579: connect: Connection refused (error 111)
I20260620 18:44:56.486119 19591 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40579 } attempt: 1
I20260620 18:44:56.487572 19594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.486957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:40579 (local address 127.0.0.1:35740)
0620 18:44:56.487213 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:56.487374 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 18:44:56.488693 19594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.488309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:40579 (local address 127.0.0.1:35756)
0620 18:44:56.488472 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:56.488572 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 18:44:56.488979 19591 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.225.188:40579: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40579: connect: Connection refused (error 111)
I20260620 18:44:56.557236 19591 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40579 } attempt: 2
I20260620 18:44:56.558485 19594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.557895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:40579 (local address 127.0.0.1:35760)
0620 18:44:56.558129 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:56.558267 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 18:44:56.559526 19594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.559130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:40579 (local address 127.0.0.1:35776)
0620 18:44:56.559290 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:56.559385 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 18:44:56.559759 19591 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.225.188:40579: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40579: connect: Connection refused (error 111)
I20260620 18:44:56.579726 19517 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.225.189" port: 36121 } attempt: 5
Jun 20 18:44:56 dist-test-slave-k6lg krb5kdc[19437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981095, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for kudu/127.5.225.189@KRBTEST.COM
I20260620 18:44:56.593729 19596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.580683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54716 (local address 127.5.225.189:36121)
0620 18:44:56.580946 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:56.580953 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:56.580978 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:44:56.581049 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:56.581054 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:56.581140 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:56.581251 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:56.582058 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.582821 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.584003 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.584461 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.584595 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:56.588064 (+ 3469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:56.588069 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:56.588072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:56.588105 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:56.590859 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:56.591803 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:56.591806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:56.591809 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:56.591948 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:56.592422 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:56.592425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:56.592427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:56.592694 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:56.592800 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:56.593325 (+ 525us) server_negotiation.cc:299] Negotiation successful
0620 18:44:56.593512 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260620 18:44:56.594043 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.580577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:36121 (local address 127.0.0.1:54716)
0620 18:44:56.580774 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:56.580798 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:56.580936 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:56.581287 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:56.581292 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:56.581699 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:56.581895 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:56.581906 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.582975 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.582983 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:56.583843 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:56.583856 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.584612 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.584618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:56.584813 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:56.585530 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:56.585548 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:56.587853 (+ 2305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:56.591027 (+ 3174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:56.591035 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:56.591040 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:56.591680 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:56.592073 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:56.592080 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:56.592085 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:56.592289 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:56.592825 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:56.592830 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:56.593040 (+ 210us) client_negotiation.cc:769] Sending connection context
0620 18:44:56.593235 (+ 195us) client_negotiation.cc:240] Negotiation successful
0620 18:44:56.593601 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 18:44:56.595173 19517 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40579 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:44:56.596457 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.595965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:40579 (local address 127.0.0.1:35784)
0620 18:44:56.596152 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:56.596244 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 18:44:56.597450 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.597079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:40579 (local address 127.0.0.1:35800)
0620 18:44:56.597234 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:56.597337 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 18:44:56.597669 19517 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.225.188:40579: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40579: connect: Connection refused (error 111)
I20260620 18:44:56.648924 19517 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40579 } attempt: 1
I20260620 18:44:56.650354 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.649621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:40579 (local address 127.0.0.1:35802)
0620 18:44:56.649898 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:56.650056 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 18:44:56.650980 19591 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40579 } attempt: 3
I20260620 18:44:56.651456 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.651068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:40579 (local address 127.0.0.1:35816)
0620 18:44:56.651222 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:56.651338 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 18:44:56.651702 19517 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.225.188:40579: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40579: connect: Connection refused (error 111)
I20260620 18:44:56.652067 19594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.651527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:40579 (local address 127.0.0.1:35820)
0620 18:44:56.651810 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:56.651898 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260620 18:44:56.653295 19594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.652734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:40579 (local address 127.0.0.1:35836)
0620 18:44:56.653015 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:56.653122 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
W20260620 18:44:56.653584 19591 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.225.188:40579: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40579: connect: Connection refused (error 111)
I20260620 18:44:56.719969 19517 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40579 } attempt: 2
I20260620 18:44:56.721405 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.720740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:40579 (local address 127.0.0.1:35852)
0620 18:44:56.721019 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:56.721179 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 18:44:56.722606 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.722148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:40579 (local address 127.0.0.1:35860)
0620 18:44:56.722359 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:56.722467 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 18:44:56.722846 19517 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.225.188:40579: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40579: connect: Connection refused (error 111)
W20260620 18:44:56.763203 19598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:56.763494 19598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:56.763643 19598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:56.770862 19598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:44:56.770962 19598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:56.771049 19598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:56.771153 19598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:44:56.771234 19598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:44:56.781008 19598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:34127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:46693,127.5.225.189:36121,127.5.225.188:40579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.188.keytab
--principal=kudu/127.5.225.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.188:40579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.225.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:44:56.782418 19598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:56.783797 19598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:56.795399 19603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:56.795352 19604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:56.795631 19606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:56.796476 19598 server_base.cc:1034] running on GCE node
I20260620 18:44:56.796777 19591 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40579 } attempt: 4
I20260620 18:44:56.797317 19598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:56.797880 19594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.797384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:40579 (local address 127.0.0.1:35866)
0620 18:44:56.797622 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:56.797700 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 18:44:56.798846 19598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:56.798902 19594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.798542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:40579 (local address 127.0.0.1:35872)
0620 18:44:56.798681 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:56.798774 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 18:44:56.799129 19591 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.225.188:40579: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40579: connect: Connection refused (error 111)
I20260620 18:44:56.800094 19598 hybrid_clock.cc:648] HybridClock initialized: now 1781981096800044 us; error 39 us; skew 500 ppm
I20260620 18:44:56.800447 19598 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:56 dist-test-slave-k6lg krb5kdc[19437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981096, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:56.803558 19598 init.cc:377] Logged in from keytab as kudu/127.5.225.188@KRBTEST.COM (short username kudu)
I20260620 18:44:56.805059 19598 webserver.cc:460] Webserver started at http://127.5.225.188:32923/ using document root <none> and password file <none>
I20260620 18:44:56.805708 19598 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:56.805809 19598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:56.806061 19598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:56.808972 19598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-2/data/instance:
uuid: "0d31f9e1b85b4301bfef8835fb73bea1"
format_stamp: "Formatted at 2026-06-20 18:44:56 on dist-test-slave-k6lg"
server_key: "0b88fec6a73cbc4de2bbae67b10c3eec"
server_key_iv: "4fc3c785660347ef6c9b205c56754b7e"
server_key_version: "encryptionkey@0"
I20260620 18:44:56.809700 19598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-2/wal/instance:
uuid: "0d31f9e1b85b4301bfef8835fb73bea1"
format_stamp: "Formatted at 2026-06-20 18:44:56 on dist-test-slave-k6lg"
server_key: "0b88fec6a73cbc4de2bbae67b10c3eec"
server_key_iv: "4fc3c785660347ef6c9b205c56754b7e"
server_key_version: "encryptionkey@0"
I20260620 18:44:56.814832 19598 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 18:44:56.818735 19613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:56.821287 19598 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 18:44:56.821519 19598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-2/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-2/wal
uuid: "0d31f9e1b85b4301bfef8835fb73bea1"
format_stamp: "Formatted at 2026-06-20 18:44:56 on dist-test-slave-k6lg"
server_key: "0b88fec6a73cbc4de2bbae67b10c3eec"
server_key_iv: "4fc3c785660347ef6c9b205c56754b7e"
server_key_version: "encryptionkey@0"
I20260620 18:44:56.821702 19598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:56.829066 19517 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40579 } attempt: 3
I20260620 18:44:56.830307 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.829689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:40579 (local address 127.0.0.1:35888)
0620 18:44:56.829982 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:56.830071 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260620 18:44:56.831347 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.830976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:40579 (local address 127.0.0.1:35896)
0620 18:44:56.831187 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:56.831259 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260620 18:44:56.831565 19517 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.225.188:40579: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40579: connect: Connection refused (error 111)
I20260620 18:44:56.844947 19598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:56.849536 19598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:56.849942 19598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:56.870471 19598 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.188:40579
I20260620 18:44:56.870482 19664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.188:40579 every 8 connection(s)
I20260620 18:44:56.871964 19598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-2/data/info.pb
I20260620 18:44:56.875649 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 19598
I20260620 18:44:56.875784 19665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:56.875854 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980894474392-6022-0/minicluster-data/master-2/wal/instance
I20260620 18:44:56.876410 6022 external_mini_cluster.cc:1428] Setting key 21a2d4ec8d169667c891844d9b2614c6
I20260620 18:44:56.885447 19665 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.225.190" port: 46693 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:44:56 dist-test-slave-k6lg krb5kdc[19437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.188@KRBTEST.COM
Jun 20 18:44:56 dist-test-slave-k6lg krb5kdc[19437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981096, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.188@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:44:56.901366 19668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.878621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35908 (local address 127.5.225.188:40579)
0620 18:44:56.879490 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:56.879506 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:56.879557 (+ 51us) server_negotiation.cc:407] Connection header received
0620 18:44:56.882470 (+ 2913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:56.882497 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:56.883030 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:56.883495 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:56.885262 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.886860 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.888523 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.889370 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.890088 (+ 718us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:56.894078 (+ 3990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:56.894087 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:56.894098 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:56.894134 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:56.897403 (+ 3269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:56.898078 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:56.898094 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:56.898101 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:56.898306 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:56.898700 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:56.898706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:56.898710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:56.899208 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:56.899506 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:56.899964 (+ 458us) server_negotiation.cc:299] Negotiation successful
0620 18:44:56.900370 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":132,"server-negotiator.queue_time_us":631,"thread_start_us":443,"threads_started":1}
I20260620 18:44:56.903187 19521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.888030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60658 (local address 127.5.225.190:46693)
0620 18:44:56.888229 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:56.888236 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:56.888993 (+ 757us) server_negotiation.cc:407] Connection header received
0620 18:44:56.889160 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:56.889166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:56.889247 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:56.889360 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:56.890727 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.891399 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.893164 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.893600 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.893739 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:56.897604 (+ 3865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:56.897609 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:56.897613 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:56.897645 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:56.900385 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:56.901065 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:56.901074 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:56.901078 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:56.901214 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:56.901656 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:56.901662 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:56.901665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:56.901930 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:56.902092 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:56.902603 (+ 511us) server_negotiation.cc:299] Negotiation successful
0620 18:44:56.902823 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 18:44:56.903241 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.888059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:46693 (local address 127.0.0.1:60658)
0620 18:44:56.888836 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:56.888887 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:56.889038 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:56.889438 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:56.889445 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:56.890056 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:56.890568 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:56.890582 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.891574 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.891582 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:56.893009 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:56.893021 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.893839 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.893846 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:56.894007 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:56.894736 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:56.894760 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:56.897413 (+ 2653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:56.900520 (+ 3107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:56.900530 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:56.900537 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:56.900944 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:56.901328 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:56.901332 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:56.901336 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:56.901558 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:56.902071 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:56.902079 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:56.902311 (+ 232us) client_negotiation.cc:769] Sending connection context
0620 18:44:56.902492 (+ 181us) client_negotiation.cc:240] Negotiation successful
0620 18:44:56.902816 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"mutex_wait_us":21,"thread_start_us":407,"threads_started":1}
I20260620 18:44:56.904758 19665 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.225.189" port: 36121 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:44:56 dist-test-slave-k6lg krb5kdc[19437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981096, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.188@KRBTEST.COM for kudu/127.5.225.189@KRBTEST.COM
Jun 20 18:44:56 dist-test-slave-k6lg krb5kdc[19437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:44:56.917738 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.905357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:36121 (local address 127.0.0.1:54722)
0620 18:44:56.905496 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:56.905514 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:56.905597 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:56.905994 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:56.906000 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:56.906508 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:56.906699 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:56.906712 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.907767 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.907775 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:56.908637 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:56.908648 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.909347 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.909353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:56.909571 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:56.910111 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:56.910132 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:56.912126 (+ 1994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:56.915394 (+ 3268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:56.915403 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:56.915408 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:56.915923 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:56.916311 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:56.916317 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:56.916322 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:56.916485 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:56.916985 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:56.916990 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:56.917164 (+ 174us) client_negotiation.cc:769] Sending connection context
0620 18:44:56.917328 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 18:44:56.917519 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 18:44:56.918197 19596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.905540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54722 (local address 127.5.225.189:36121)
0620 18:44:56.905702 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:56.905706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:56.905720 (+ 14us) server_negotiation.cc:407] Connection header received
0620 18:44:56.905766 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:56.905770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:56.905823 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:56.905933 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:56.906858 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.907528 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.908780 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.909198 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.909327 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:56.912313 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:56.912317 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:56.912321 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:56.912358 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:56.915259 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:56.916049 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:56.916053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:56.916055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:56.916192 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:56.916613 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:56.916617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:56.916619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:56.916858 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:56.916968 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:56.917543 (+ 575us) server_negotiation.cc:299] Negotiation successful
0620 18:44:56.917756 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:44:56.919190 19665 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40579 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:44:56.920214 19521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.904601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60670 (local address 127.5.225.190:46693)
0620 18:44:56.904772 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:56.904780 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:56.904800 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:44:56.904995 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:56.905001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:56.905069 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:56.905181 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:56.907170 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.907901 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.909157 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.909587 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.909714 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:56.914107 (+ 4393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:56.914115 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:56.914122 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:56.914188 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:56.917137 (+ 2949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:56.918033 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:56.918039 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:56.918043 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:56.918225 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:56.918688 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:56.918694 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:56.918699 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:56.919076 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:56.919310 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:56.919648 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 18:44:56.919896 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 18:44:56 dist-test-slave-k6lg krb5kdc[19437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981096, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.188@KRBTEST.COM for kudu/127.5.225.188@KRBTEST.COM
Jun 20 18:44:56 dist-test-slave-k6lg krb5kdc[19437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.189@KRBTEST.COM
I20260620 18:44:56.933223 19668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.920028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35910 (local address 127.5.225.188:40579)
0620 18:44:56.920210 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:56.920217 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:56.920332 (+ 115us) server_negotiation.cc:407] Connection header received
0620 18:44:56.920438 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:56.920443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:56.920511 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:56.920598 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:56.921166 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.922214 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.923510 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.924007 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.924095 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:56.928023 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:56.928029 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:56.928034 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:56.928069 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:56.930638 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:56.931233 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:56.931241 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:56.931244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:56.931361 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:56.931718 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:56.931723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:56.931727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:56.932092 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:56.932245 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:56.932593 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 18:44:56.932849 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":78}
I20260620 18:44:56.933332 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.920067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:40579 (local address 127.0.0.1:35910)
0620 18:44:56.920209 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:56.920227 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:56.920332 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:56.920619 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:56.920623 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:56.920931 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:56.921051 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:56.921058 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.922320 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.922325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:56.923362 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:56.923373 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.924144 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.924151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:56.924333 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:56.924905 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:56.924925 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:56.927844 (+ 2919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:56.930755 (+ 2911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:56.930763 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:56.930767 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:56.931121 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:56.931468 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:56.931474 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:56.931479 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:56.931609 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:56.932209 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:56.932214 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:56.932363 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 18:44:56.932521 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 18:44:56.933002 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":158}
I20260620 18:44:56.936694 19596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.921885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54732 (local address 127.5.225.189:36121)
0620 18:44:56.922051 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:56.922056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:56.922072 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:44:56.922256 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:56.922261 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:56.922331 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:56.922426 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:56.923650 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.924497 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.926012 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.926846 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.926966 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:56.931298 (+ 4332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:56.931303 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:56.931306 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:56.931335 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:56.933966 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:56.934660 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:56.934667 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:56.934670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:56.934836 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:56.935233 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:56.935238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:56.935241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:56.935613 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:56.935897 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:56.936118 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 18:44:56.936368 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:44:56.939208 19665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d31f9e1b85b4301bfef8835fb73bea1: Bootstrap starting.
I20260620 18:44:56.941553 6022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:44:56.942867 19665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d31f9e1b85b4301bfef8835fb73bea1: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:56.943915 19665 log.cc:824] T 00000000000000000000000000000000 P 0d31f9e1b85b4301bfef8835fb73bea1: Log is configured to *not* fsync() on all Append() calls
I20260620 18:44:56.946753 19665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d31f9e1b85b4301bfef8835fb73bea1: No bootstrap required, opened a new log
I20260620 18:44:56.957681 19665 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d31f9e1b85b4301bfef8835fb73bea1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b40cedf94f8499fa151f6501d7f5083" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 46693 } } peers { permanent_uuid: "aa4fc73911854935bb5eaa622c93914a" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 36121 } } peers { permanent_uuid: "0d31f9e1b85b4301bfef8835fb73bea1" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40579 } }
I20260620 18:44:56.958257 19665 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d31f9e1b85b4301bfef8835fb73bea1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:56.958420 19665 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d31f9e1b85b4301bfef8835fb73bea1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d31f9e1b85b4301bfef8835fb73bea1, State: Initialized, Role: FOLLOWER
I20260620 18:44:56.959646 19665 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d31f9e1b85b4301bfef8835fb73bea1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b40cedf94f8499fa151f6501d7f5083" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 46693 } } peers { permanent_uuid: "aa4fc73911854935bb5eaa622c93914a" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 36121 } } peers { permanent_uuid: "0d31f9e1b85b4301bfef8835fb73bea1" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40579 } }
I20260620 18:44:56.963915 19665 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d31f9e1b85b4301bfef8835fb73bea1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:44:56.966239 19678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d31f9e1b85b4301bfef8835fb73bea1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b40cedf94f8499fa151f6501d7f5083" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 46693 } } peers { permanent_uuid: "aa4fc73911854935bb5eaa622c93914a" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 36121 } } peers { permanent_uuid: "0d31f9e1b85b4301bfef8835fb73bea1" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40579 } } }
I20260620 18:44:56.966773 19678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d31f9e1b85b4301bfef8835fb73bea1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:44:56.968080 19521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.948878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60672 (local address 127.5.225.190:46693)
0620 18:44:56.949353 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:56.949360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:56.949895 (+ 535us) server_negotiation.cc:407] Connection header received
0620 18:44:56.950205 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:56.950212 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:56.950290 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:56.950379 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:56.953171 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.954319 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.956795 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.957442 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.957557 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:56.961351 (+ 3794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:56.961361 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:56.961367 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:56.961430 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:56.964995 (+ 3565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:56.965833 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:56.965840 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:56.965844 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:56.966013 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:56.966566 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:56.966572 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:56.966576 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:56.966952 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:56.967215 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:56.967530 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 18:44:56.967749 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334}
I20260620 18:44:56.971536 19596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.950092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54738 (local address 127.5.225.189:36121)
0620 18:44:56.950335 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:56.950343 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:56.953663 (+ 3320us) server_negotiation.cc:407] Connection header received
0620 18:44:56.954091 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:56.954096 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:56.954200 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:56.954264 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:56.956525 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.957589 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.959799 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.960527 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.960633 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:56.964273 (+ 3640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:56.964281 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:56.964286 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:56.964326 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:56.967836 (+ 3510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:56.968765 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:56.968772 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:56.968778 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:56.968944 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:56.969430 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:56.969435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:56.969438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:56.969806 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:56.970042 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:56.971131 (+ 1089us) server_negotiation.cc:299] Negotiation successful
0620 18:44:56.971306 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260620 18:44:56.971552 19668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.949147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35918 (local address 127.5.225.188:40579)
0620 18:44:56.949566 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:56.949573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:56.953192 (+ 3619us) server_negotiation.cc:407] Connection header received
0620 18:44:56.953377 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:56.953383 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:56.953459 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:56.953556 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:56.955830 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.956934 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.958989 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.959705 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.959807 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:56.963616 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:56.963622 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:56.963626 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:56.963661 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:56.967401 (+ 3740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:56.968434 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:56.968443 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:56.968449 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:56.968656 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:56.969191 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:56.969199 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:56.969205 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:56.969699 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:56.970000 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:56.970744 (+ 744us) server_negotiation.cc:299] Negotiation successful
0620 18:44:56.971042 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275}
W20260620 18:44:56.982527 19689 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0d31f9e1b85b4301bfef8835fb73bea1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:44:56.982666 19689 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:44:56.979197 19598 master_runner.cc:186] Error getting master registration for 127.5.225.190:46693: OK, instance_id { permanent_uuid: "4b40cedf94f8499fa151f6501d7f5083" instance_seqno: 1781981095870655 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:44:56.988265 19598 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 18:44:57.008797 19517 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40579 } attempt: 4
Jun 20 18:44:57 dist-test-slave-k6lg krb5kdc[19437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981095, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for kudu/127.5.225.188@KRBTEST.COM
I20260620 18:44:57.024412 19668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:57.009505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35926 (local address 127.5.225.188:40579)
0620 18:44:57.009668 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:57.009676 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:57.009699 (+ 23us) server_negotiation.cc:407] Connection header received
0620 18:44:57.009914 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:57.009919 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:57.010009 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:57.010117 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:57.011164 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:57.012138 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:57.013877 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:57.014377 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:57.014471 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:57.018333 (+ 3862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:57.018339 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:57.018344 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:57.018383 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:57.021640 (+ 3257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:57.022425 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:57.022446 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:57.022452 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:57.022629 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:57.023076 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:57.023080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:57.023082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:57.023369 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:57.023554 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:57.023971 (+ 417us) server_negotiation.cc:299] Negotiation successful
0620 18:44:57.024150 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:44:57.024529 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:57.009434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:40579 (local address 127.0.0.1:35926)
0620 18:44:57.009634 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:57.009655 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:57.009780 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:57.010183 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:57.010191 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:57.010770 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:57.010993 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:57.011006 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:57.012283 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:57.012290 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:57.013735 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:57.013751 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:57.014519 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:57.014526 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:57.014717 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:57.015358 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:57.015403 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:57.018113 (+ 2710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:57.021757 (+ 3644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:57.021763 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:57.021767 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:57.022311 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:57.022723 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:57.022728 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:57.022732 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:57.022915 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:57.023532 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:57.023537 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:57.023698 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 18:44:57.023843 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 18:44:57.024129 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 18:44:57.029142 19517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b40cedf94f8499fa151f6501d7f5083: Bootstrap starting.
I20260620 18:44:57.032749 19517 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b40cedf94f8499fa151f6501d7f5083: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:57.033985 19517 log.cc:824] T 00000000000000000000000000000000 P 4b40cedf94f8499fa151f6501d7f5083: Log is configured to *not* fsync() on all Append() calls
I20260620 18:44:57.036793 19517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b40cedf94f8499fa151f6501d7f5083: No bootstrap required, opened a new log
I20260620 18:44:57.045115 19517 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b40cedf94f8499fa151f6501d7f5083 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b40cedf94f8499fa151f6501d7f5083" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 46693 } } peers { permanent_uuid: "aa4fc73911854935bb5eaa622c93914a" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 36121 } } peers { permanent_uuid: "0d31f9e1b85b4301bfef8835fb73bea1" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40579 } }
I20260620 18:44:57.045467 19517 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b40cedf94f8499fa151f6501d7f5083 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:57.045599 19517 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b40cedf94f8499fa151f6501d7f5083 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b40cedf94f8499fa151f6501d7f5083, State: Initialized, Role: FOLLOWER
I20260620 18:44:57.046198 19517 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b40cedf94f8499fa151f6501d7f5083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b40cedf94f8499fa151f6501d7f5083" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 46693 } } peers { permanent_uuid: "aa4fc73911854935bb5eaa622c93914a" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 36121 } } peers { permanent_uuid: "0d31f9e1b85b4301bfef8835fb73bea1" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40579 } }
I20260620 18:44:57.047215 19692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b40cedf94f8499fa151f6501d7f5083 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b40cedf94f8499fa151f6501d7f5083" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 46693 } } peers { permanent_uuid: "aa4fc73911854935bb5eaa622c93914a" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 36121 } } peers { permanent_uuid: "0d31f9e1b85b4301bfef8835fb73bea1" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40579 } } }
I20260620 18:44:57.047600 19692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b40cedf94f8499fa151f6501d7f5083 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:44:57.048421 19517 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b40cedf94f8499fa151f6501d7f5083 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 18:44:57.059784 19703 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4b40cedf94f8499fa151f6501d7f5083: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:44:57.059991 19703 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:44:57.058243 19450 master_runner.cc:186] Error getting master registration for 127.5.225.189:36121: OK, instance_id { permanent_uuid: "aa4fc73911854935bb5eaa622c93914a" instance_seqno: 1781981096366415 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:44:57.066628 19450 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 18:44:57.098397 19591 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40579 } attempt: 5
W20260620 18:44:57.102377 19619 tablet.cc:2369] T 00000000000000000000000000000000 P 0d31f9e1b85b4301bfef8835fb73bea1: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 18:44:57 dist-test-slave-k6lg krb5kdc[19437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981096, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.189@KRBTEST.COM for kudu/127.5.225.188@KRBTEST.COM
W20260620 18:44:57.110073 19471 tablet.cc:2369] T 00000000000000000000000000000000 P 4b40cedf94f8499fa151f6501d7f5083: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:44:57.113845 19594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:57.099167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:40579 (local address 127.0.0.1:35936)
0620 18:44:57.099458 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:57.099477 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:57.099633 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:57.099989 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:57.099996 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:57.100465 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:57.100625 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:57.100634 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:57.101930 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:57.101936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:57.103253 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:57.103267 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:57.104506 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:57.104513 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:57.104754 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:57.105461 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:57.105486 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:57.107617 (+ 2131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:57.111315 (+ 3698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:57.111321 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:57.111334 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:57.111742 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:57.112126 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:57.112132 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:57.112136 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:57.112287 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:57.112810 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:57.112816 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:57.112974 (+ 158us) client_negotiation.cc:769] Sending connection context
0620 18:44:57.113119 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 18:44:57.113426 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184}
I20260620 18:44:57.113863 19668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:57.099229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35936 (local address 127.5.225.188:40579)
0620 18:44:57.099417 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:57.099424 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:57.099563 (+ 139us) server_negotiation.cc:407] Connection header received
0620 18:44:57.099776 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:57.099781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:57.099847 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:57.099965 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:57.100783 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:57.101724 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:57.103431 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:57.104096 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:57.104221 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:57.107814 (+ 3593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:57.107820 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:57.107823 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:57.107853 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:57.111181 (+ 3328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:57.111873 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:57.111878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:57.111882 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:57.112012 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:57.112394 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:57.112400 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:57.112403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:57.112684 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:57.112864 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:57.113224 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 18:44:57.113464 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:44:57.118798 19591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa4fc73911854935bb5eaa622c93914a: Bootstrap starting.
I20260620 18:44:57.122840 19591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa4fc73911854935bb5eaa622c93914a: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:57.124297 19591 log.cc:824] T 00000000000000000000000000000000 P aa4fc73911854935bb5eaa622c93914a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:44:57.127380 19591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa4fc73911854935bb5eaa622c93914a: No bootstrap required, opened a new log
I20260620 18:44:57.134644 19591 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa4fc73911854935bb5eaa622c93914a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b40cedf94f8499fa151f6501d7f5083" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 46693 } } peers { permanent_uuid: "aa4fc73911854935bb5eaa622c93914a" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 36121 } } peers { permanent_uuid: "0d31f9e1b85b4301bfef8835fb73bea1" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40579 } }
I20260620 18:44:57.135011 19591 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa4fc73911854935bb5eaa622c93914a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:57.135161 19591 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa4fc73911854935bb5eaa622c93914a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa4fc73911854935bb5eaa622c93914a, State: Initialized, Role: FOLLOWER
I20260620 18:44:57.135794 19591 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa4fc73911854935bb5eaa622c93914a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b40cedf94f8499fa151f6501d7f5083" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 46693 } } peers { permanent_uuid: "aa4fc73911854935bb5eaa622c93914a" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 36121 } } peers { permanent_uuid: "0d31f9e1b85b4301bfef8835fb73bea1" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40579 } }
I20260620 18:44:57.136736 19706 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa4fc73911854935bb5eaa622c93914a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b40cedf94f8499fa151f6501d7f5083" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 46693 } } peers { permanent_uuid: "aa4fc73911854935bb5eaa622c93914a" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 36121 } } peers { permanent_uuid: "0d31f9e1b85b4301bfef8835fb73bea1" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40579 } } }
I20260620 18:44:57.137151 19706 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa4fc73911854935bb5eaa622c93914a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:44:57.137985 19591 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa4fc73911854935bb5eaa622c93914a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:44:57.160406 19524 master_runner.cc:263] No leader master found from master aa4fc73911854935bb5eaa622c93914a
I20260620 18:44:57.160622 19524 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 18:44:57.169895 19692 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b40cedf94f8499fa151f6501d7f5083 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:44:57.170456 19692 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b40cedf94f8499fa151f6501d7f5083 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b40cedf94f8499fa151f6501d7f5083" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 46693 } } peers { permanent_uuid: "aa4fc73911854935bb5eaa622c93914a" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 36121 } } peers { permanent_uuid: "0d31f9e1b85b4301bfef8835fb73bea1" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40579 } }
I20260620 18:44:57.172084 19692 leader_election.cc:288] T 00000000000000000000000000000000 P 4b40cedf94f8499fa151f6501d7f5083 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa4fc73911854935bb5eaa622c93914a (127.5.225.189:36121), 0d31f9e1b85b4301bfef8835fb73bea1 (127.5.225.188:40579)
W20260620 18:44:57.173121 19717 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aa4fc73911854935bb5eaa622c93914a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:44:57.173321 19717 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:44:57.173847 19566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4b40cedf94f8499fa151f6501d7f5083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa4fc73911854935bb5eaa622c93914a" is_pre_election: true
I20260620 18:44:57.173856 19640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4b40cedf94f8499fa151f6501d7f5083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d31f9e1b85b4301bfef8835fb73bea1" is_pre_election: true
I20260620 18:44:57.174494 19640 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0d31f9e1b85b4301bfef8835fb73bea1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b40cedf94f8499fa151f6501d7f5083 in term 0.
I20260620 18:44:57.174494 19566 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aa4fc73911854935bb5eaa622c93914a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b40cedf94f8499fa151f6501d7f5083 in term 0.
I20260620 18:44:57.175150 19469 leader_election.cc:302] T 00000000000000000000000000000000 P 4b40cedf94f8499fa151f6501d7f5083 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d31f9e1b85b4301bfef8835fb73bea1, 4b40cedf94f8499fa151f6501d7f5083; no voters:
I20260620 18:44:57.175911 19692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b40cedf94f8499fa151f6501d7f5083 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:44:57.176061 19692 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b40cedf94f8499fa151f6501d7f5083 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:44:57.176164 19692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b40cedf94f8499fa151f6501d7f5083 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:57.179620 19692 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b40cedf94f8499fa151f6501d7f5083 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b40cedf94f8499fa151f6501d7f5083" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 46693 } } peers { permanent_uuid: "aa4fc73911854935bb5eaa622c93914a" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 36121 } } peers { permanent_uuid: "0d31f9e1b85b4301bfef8835fb73bea1" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40579 } }
I20260620 18:44:57.180878 19566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4b40cedf94f8499fa151f6501d7f5083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa4fc73911854935bb5eaa622c93914a"
I20260620 18:44:57.181103 19566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa4fc73911854935bb5eaa622c93914a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:57.184626 19566 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aa4fc73911854935bb5eaa622c93914a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b40cedf94f8499fa151f6501d7f5083 in term 1.
I20260620 18:44:57.185165 19468 leader_election.cc:302] T 00000000000000000000000000000000 P 4b40cedf94f8499fa151f6501d7f5083 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b40cedf94f8499fa151f6501d7f5083, aa4fc73911854935bb5eaa622c93914a; no voters:
I20260620 18:44:57.188390 19640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4b40cedf94f8499fa151f6501d7f5083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d31f9e1b85b4301bfef8835fb73bea1"
I20260620 18:44:57.188630 19640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d31f9e1b85b4301bfef8835fb73bea1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:57.190590 19719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b40cedf94f8499fa151f6501d7f5083 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:44:57.192142 19640 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0d31f9e1b85b4301bfef8835fb73bea1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b40cedf94f8499fa151f6501d7f5083 in term 1.
I20260620 18:44:57.196381 19692 leader_election.cc:288] T 00000000000000000000000000000000 P 4b40cedf94f8499fa151f6501d7f5083 [CANDIDATE]: Term 1 election: Requested vote from peers aa4fc73911854935bb5eaa622c93914a (127.5.225.189:36121), 0d31f9e1b85b4301bfef8835fb73bea1 (127.5.225.188:40579)
I20260620 18:44:57.198487 19719 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b40cedf94f8499fa151f6501d7f5083 [term 1 LEADER]: Becoming Leader. State: Replica: 4b40cedf94f8499fa151f6501d7f5083, State: Running, Role: LEADER
I20260620 18:44:57.199208 19719 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b40cedf94f8499fa151f6501d7f5083 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b40cedf94f8499fa151f6501d7f5083" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 46693 } } peers { permanent_uuid: "aa4fc73911854935bb5eaa622c93914a" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 36121 } } peers { permanent_uuid: "0d31f9e1b85b4301bfef8835fb73bea1" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40579 } }
I20260620 18:44:57.220999 19692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b40cedf94f8499fa151f6501d7f5083 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b40cedf94f8499fa151f6501d7f5083. Latest consensus state: current_term: 1 leader_uuid: "4b40cedf94f8499fa151f6501d7f5083" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b40cedf94f8499fa151f6501d7f5083" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 46693 } } peers { permanent_uuid: "aa4fc73911854935bb5eaa622c93914a" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 36121 } } peers { permanent_uuid: "0d31f9e1b85b4301bfef8835fb73bea1" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40579 } } }
I20260620 18:44:57.221366 19692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b40cedf94f8499fa151f6501d7f5083 [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:57.230798 19722 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:44:57.232570 19722 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:44:57.248847 19640 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0d31f9e1b85b4301bfef8835fb73bea1 [term 1 FOLLOWER]: Refusing update from remote peer 4b40cedf94f8499fa151f6501d7f5083: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:44:57.250016 19692 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b40cedf94f8499fa151f6501d7f5083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d31f9e1b85b4301bfef8835fb73bea1" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:44:57.257592 19566 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aa4fc73911854935bb5eaa622c93914a [term 1 FOLLOWER]: Refusing update from remote peer 4b40cedf94f8499fa151f6501d7f5083: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:44:57.258591 19692 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b40cedf94f8499fa151f6501d7f5083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa4fc73911854935bb5eaa622c93914a" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 36121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:44:57.277415 19678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d31f9e1b85b4301bfef8835fb73bea1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b40cedf94f8499fa151f6501d7f5083. Latest consensus state: current_term: 1 leader_uuid: "4b40cedf94f8499fa151f6501d7f5083" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b40cedf94f8499fa151f6501d7f5083" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 46693 } } peers { permanent_uuid: "aa4fc73911854935bb5eaa622c93914a" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 36121 } } peers { permanent_uuid: "0d31f9e1b85b4301bfef8835fb73bea1" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40579 } } }
I20260620 18:44:57.277751 19678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d31f9e1b85b4301bfef8835fb73bea1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:44:57.283349 19723 mvcc.cc:205] Tried to move back new op lower bound from 7298994574320680960 to 7298994574133182464. Current Snapshot: MvccSnapshot[applied={T|T < 7298994574320680960}]
I20260620 18:44:57.284893 19692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b40cedf94f8499fa151f6501d7f5083 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b40cedf94f8499fa151f6501d7f5083" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b40cedf94f8499fa151f6501d7f5083" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 46693 } } peers { permanent_uuid: "aa4fc73911854935bb5eaa622c93914a" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 36121 } } peers { permanent_uuid: "0d31f9e1b85b4301bfef8835fb73bea1" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40579 } } }
I20260620 18:44:57.285359 19692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b40cedf94f8499fa151f6501d7f5083 [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:57.286324 19724 mvcc.cc:205] Tried to move back new op lower bound from 7298994574320680960 to 7298994574133182464. Current Snapshot: MvccSnapshot[applied={T|T < 7298994574320680960}]
I20260620 18:44:57.287079 19722 catalog_manager.cc:1347] Generated new cluster ID: f6d9d68a26624dfe87b598a2bde4ad1f
I20260620 18:44:57.287206 19722 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:44:57.289579 19706 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa4fc73911854935bb5eaa622c93914a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b40cedf94f8499fa151f6501d7f5083. Latest consensus state: current_term: 1 leader_uuid: "4b40cedf94f8499fa151f6501d7f5083" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b40cedf94f8499fa151f6501d7f5083" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 46693 } } peers { permanent_uuid: "aa4fc73911854935bb5eaa622c93914a" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 36121 } } peers { permanent_uuid: "0d31f9e1b85b4301bfef8835fb73bea1" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40579 } } }
I20260620 18:44:57.289901 19706 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa4fc73911854935bb5eaa622c93914a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:44:57.293309 19692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b40cedf94f8499fa151f6501d7f5083 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b40cedf94f8499fa151f6501d7f5083" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b40cedf94f8499fa151f6501d7f5083" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 46693 } } peers { permanent_uuid: "aa4fc73911854935bb5eaa622c93914a" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 36121 } } peers { permanent_uuid: "0d31f9e1b85b4301bfef8835fb73bea1" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40579 } } }
I20260620 18:44:57.293710 19692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b40cedf94f8499fa151f6501d7f5083 [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:57.313683 19706 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa4fc73911854935bb5eaa622c93914a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b40cedf94f8499fa151f6501d7f5083" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b40cedf94f8499fa151f6501d7f5083" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 46693 } } peers { permanent_uuid: "aa4fc73911854935bb5eaa622c93914a" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 36121 } } peers { permanent_uuid: "0d31f9e1b85b4301bfef8835fb73bea1" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40579 } } }
I20260620 18:44:57.313994 19706 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa4fc73911854935bb5eaa622c93914a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:44:57.314064 19678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d31f9e1b85b4301bfef8835fb73bea1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b40cedf94f8499fa151f6501d7f5083" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b40cedf94f8499fa151f6501d7f5083" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 46693 } } peers { permanent_uuid: "aa4fc73911854935bb5eaa622c93914a" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 36121 } } peers { permanent_uuid: "0d31f9e1b85b4301bfef8835fb73bea1" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40579 } } }
I20260620 18:44:57.314337 19678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d31f9e1b85b4301bfef8835fb73bea1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:44:57.328799 19722 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:44:57.330742 19722 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:44:57.343357 19722 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b40cedf94f8499fa151f6501d7f5083: Generated new TSK 0
I20260620 18:44:57.344555 19722 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:44:57.984593 19689 catalog_manager.cc:1259] Loaded cluster ID: f6d9d68a26624dfe87b598a2bde4ad1f
I20260620 18:44:57.984719 19689 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0d31f9e1b85b4301bfef8835fb73bea1: loading cluster ID for follower catalog manager: success
I20260620 18:44:57.987191 19689 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0d31f9e1b85b4301bfef8835fb73bea1: acquiring CA information for follower catalog manager: success
I20260620 18:44:57.989241 19689 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0d31f9e1b85b4301bfef8835fb73bea1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:44:58.175549 19717 catalog_manager.cc:1259] Loaded cluster ID: f6d9d68a26624dfe87b598a2bde4ad1f
I20260620 18:44:58.175681 19717 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aa4fc73911854935bb5eaa622c93914a: loading cluster ID for follower catalog manager: success
I20260620 18:44:58.178130 19717 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aa4fc73911854935bb5eaa622c93914a: acquiring CA information for follower catalog manager: success
I20260620 18:44:58.179961 19717 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aa4fc73911854935bb5eaa622c93914a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:44:58.496637 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 19450
I20260620 18:44:58.498018 19512 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:58.611004 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 19450
I20260620 18:44:58.633091 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 19524
I20260620 18:44:58.634395 19586 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:58.741072 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 19524
I20260620 18:44:58.761253 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 19598
I20260620 18:44:58.762540 19660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:58.870857 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 19598
2026-06-20T18:44:58Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4760 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:44:58 dist-test-slave-k6lg krb5kdc[19739](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:44:58 dist-test-slave-k6lg krb5kdc[19739](info): set up 2 sockets
Jun 20 18:44:58 dist-test-slave-k6lg krb5kdc[19739](info): commencing operation
krb5kdc: starting...
W20260620 18:44:59.974741 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.009s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:45:00 dist-test-slave-k6lg krb5kdc[19739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:45:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:45:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
I20260620 18:45:00.211508 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:44651
--webserver_interface=127.5.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:40833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:44651
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:45:00.539546 19752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:00.539850 19752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:00.539942 19752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:00.547964 19752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:45:00.548070 19752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:00.548117 19752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:00.548166 19752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:45:00.548209 19752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:45:00.558104 19752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:40833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:44651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:44651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:45:00.559410 19752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:00.560626 19752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:00.572372 19757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:00.572479 19760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:00.572340 19758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:00.573115 19752 server_base.cc:1034] running on GCE node
I20260620 18:45:00.573909 19752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:00.575364 19752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:00.576601 19752 hybrid_clock.cc:648] HybridClock initialized: now 1781981100576556 us; error 54 us; skew 500 ppm
I20260620 18:45:00.576956 19752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:45:00 dist-test-slave-k6lg krb5kdc[19739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981100, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:45:00.580180 19752 init.cc:377] Logged in from keytab as kudu/127.5.225.190@KRBTEST.COM (short username kudu)
I20260620 18:45:00.581720 19752 webserver.cc:460] Webserver started at http://127.5.225.190:40453/ using document root <none> and password file <none>
I20260620 18:45:00.582445 19752 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:00.582545 19752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:00.582799 19752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:45:00.586119 19752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "19a980cee1be4627a98071d6cd2e0f2d"
format_stamp: "Formatted at 2026-06-20 18:45:00 on dist-test-slave-k6lg"
server_key: "6809461e8c34e6a2fc9093bc6e72636a"
server_key_iv: "c0212491aa60e7b9add3c84245005eba"
server_key_version: "encryptionkey@0"
I20260620 18:45:00.587045 19752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "19a980cee1be4627a98071d6cd2e0f2d"
format_stamp: "Formatted at 2026-06-20 18:45:00 on dist-test-slave-k6lg"
server_key: "6809461e8c34e6a2fc9093bc6e72636a"
server_key_iv: "c0212491aa60e7b9add3c84245005eba"
server_key_version: "encryptionkey@0"
I20260620 18:45:00.592085 19752 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:45:00.595546 19767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:00.596815 19752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:45:00.597007 19752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "19a980cee1be4627a98071d6cd2e0f2d"
format_stamp: "Formatted at 2026-06-20 18:45:00 on dist-test-slave-k6lg"
server_key: "6809461e8c34e6a2fc9093bc6e72636a"
server_key_iv: "c0212491aa60e7b9add3c84245005eba"
server_key_version: "encryptionkey@0"
I20260620 18:45:00.597198 19752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:00.615051 19752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:00.619359 19752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:00.619645 19752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:00.638906 19752 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:44651
I20260620 18:45:00.638938 19818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:44651 every 8 connection(s)
I20260620 18:45:00.640285 19752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:45:00.643517 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 19752
I20260620 18:45:00.643733 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:45:00.644448 19819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:45:00.644634 6022 external_mini_cluster.cc:1428] Setting key 42236c34a61ecc88d6bab99644584940
I20260620 18:45:00.657025 19819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19a980cee1be4627a98071d6cd2e0f2d: Bootstrap starting.
Jun 20 18:45:00 dist-test-slave-k6lg krb5kdc[19739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:45:00.660440 19819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19a980cee1be4627a98071d6cd2e0f2d: Neither blocks nor log segments found. Creating new log.
I20260620 18:45:00.661631 19819 log.cc:824] T 00000000000000000000000000000000 P 19a980cee1be4627a98071d6cd2e0f2d: Log is configured to *not* fsync() on all Append() calls
I20260620 18:45:00.664363 19819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19a980cee1be4627a98071d6cd2e0f2d: No bootstrap required, opened a new log
I20260620 18:45:00.667212 19822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:00.646579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32942 (local address 127.5.225.190:44651)
0620 18:45:00.647390 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:00.647401 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:00.647459 (+ 58us) server_negotiation.cc:407] Connection header received
0620 18:45:00.649332 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:00.649352 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:00.649797 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:00.650249 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:00.651771 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:00.653010 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:00.654424 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:00.655123 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:00.655326 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:00.659689 (+ 4363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:00.659697 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:00.659710 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:00.659746 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:00.663028 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:00.663728 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:00.663734 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:00.663740 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:00.663915 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:00.664335 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:00.664339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:00.664341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:00.664950 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:00.665272 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:00.665687 (+ 415us) server_negotiation.cc:299] Negotiation successful
0620 18:45:00.666151 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":387,"threads_started":1}
I20260620 18:45:00.671051 6022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:45:00.672451 19819 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19a980cee1be4627a98071d6cd2e0f2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a980cee1be4627a98071d6cd2e0f2d" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 44651 } }
I20260620 18:45:00.672785 19819 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19a980cee1be4627a98071d6cd2e0f2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:45:00.672885 19819 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19a980cee1be4627a98071d6cd2e0f2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19a980cee1be4627a98071d6cd2e0f2d, State: Initialized, Role: FOLLOWER
I20260620 18:45:00.673542 19819 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19a980cee1be4627a98071d6cd2e0f2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a980cee1be4627a98071d6cd2e0f2d" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 44651 } }
I20260620 18:45:00.673805 19819 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19a980cee1be4627a98071d6cd2e0f2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:45:00.673946 19819 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19a980cee1be4627a98071d6cd2e0f2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:45:00.674130 19819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19a980cee1be4627a98071d6cd2e0f2d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:45:00.676280 19819 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19a980cee1be4627a98071d6cd2e0f2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a980cee1be4627a98071d6cd2e0f2d" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 44651 } }
I20260620 18:45:00.676697 19819 leader_election.cc:302] T 00000000000000000000000000000000 P 19a980cee1be4627a98071d6cd2e0f2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19a980cee1be4627a98071d6cd2e0f2d; no voters:
I20260620 18:45:00.677255 19819 leader_election.cc:288] T 00000000000000000000000000000000 P 19a980cee1be4627a98071d6cd2e0f2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:45:00.677382 19824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19a980cee1be4627a98071d6cd2e0f2d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:45:00.678071 19824 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19a980cee1be4627a98071d6cd2e0f2d [term 1 LEADER]: Becoming Leader. State: Replica: 19a980cee1be4627a98071d6cd2e0f2d, State: Running, Role: LEADER
I20260620 18:45:00.678642 19824 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19a980cee1be4627a98071d6cd2e0f2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a980cee1be4627a98071d6cd2e0f2d" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 44651 } }
I20260620 18:45:00.679279 19819 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19a980cee1be4627a98071d6cd2e0f2d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:45:00.681308 19826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19a980cee1be4627a98071d6cd2e0f2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19a980cee1be4627a98071d6cd2e0f2d. Latest consensus state: current_term: 1 leader_uuid: "19a980cee1be4627a98071d6cd2e0f2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a980cee1be4627a98071d6cd2e0f2d" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 44651 } } }
I20260620 18:45:00.681310 19825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19a980cee1be4627a98071d6cd2e0f2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19a980cee1be4627a98071d6cd2e0f2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a980cee1be4627a98071d6cd2e0f2d" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 44651 } } }
I20260620 18:45:00.681751 19825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19a980cee1be4627a98071d6cd2e0f2d [sys.catalog]: This master's current role is: LEADER
I20260620 18:45:00.682372 19826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19a980cee1be4627a98071d6cd2e0f2d [sys.catalog]: This master's current role is: LEADER
I20260620 18:45:00.683157 19829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:45:00.688266 19829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:45:00.697152 19829 catalog_manager.cc:1347] Generated new cluster ID: 2d8e9359393a407192c52d0a037d4b00
I20260620 18:45:00.697256 19829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:45:00.719897 19829 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:45:00.721199 19829 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:45:00.731102 19829 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19a980cee1be4627a98071d6cd2e0f2d: Generated new TSK 0
I20260620 18:45:00.731928 19829 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:45:00.783784 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 19752
I20260620 18:45:00.785089 19814 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:00.883144 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 19752
2026-06-20T18:45:00Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2006 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:45:00 dist-test-slave-k6lg krb5kdc[19847](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:45:00 dist-test-slave-k6lg krb5kdc[19847](info): set up 2 sockets
Jun 20 18:45:00 dist-test-slave-k6lg krb5kdc[19847](info): commencing operation
krb5kdc: starting...
W20260620 18:45:01.986778 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.007s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:45:02 dist-test-slave-k6lg krb5kdc[19847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:45:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:45:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
I20260620 18:45:02.223727 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:36571
--webserver_interface=127.5.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:33725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:36571
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:45:02.561621 19860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:02.561940 19860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:02.562108 19860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:02.569710 19860 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:45:02.569849 19860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:02.569931 19860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:02.570024 19860 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:45:02.570111 19860 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:45:02.580156 19860 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:33725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:36571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:36571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:45:02.581580 19860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:02.582990 19860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:02.594856 19866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:02.594910 19865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:02.594889 19868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:02.596428 19860 server_base.cc:1034] running on GCE node
I20260620 18:45:02.597214 19860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:02.598745 19860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:02.599946 19860 hybrid_clock.cc:648] HybridClock initialized: now 1781981102599884 us; error 54 us; skew 500 ppm
I20260620 18:45:02.600339 19860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:45:02 dist-test-slave-k6lg krb5kdc[19847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981102, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:45:02.603636 19860 init.cc:377] Logged in from keytab as kudu/127.5.225.190@KRBTEST.COM (short username kudu)
I20260620 18:45:02.605223 19860 webserver.cc:460] Webserver started at http://127.5.225.190:44363/ using document root <none> and password file <none>
I20260620 18:45:02.605898 19860 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:02.606015 19860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:02.606353 19860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:45:02.609486 19860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "0db553c30c5c482090e33c6691ea0cf1"
format_stamp: "Formatted at 2026-06-20 18:45:02 on dist-test-slave-k6lg"
server_key: "377e7c46fcd3879fc81706dbd2a3a120"
server_key_iv: "a611f987e6e52c3baa4acc419608205b"
server_key_version: "encryptionkey@0"
I20260620 18:45:02.610314 19860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "0db553c30c5c482090e33c6691ea0cf1"
format_stamp: "Formatted at 2026-06-20 18:45:02 on dist-test-slave-k6lg"
server_key: "377e7c46fcd3879fc81706dbd2a3a120"
server_key_iv: "a611f987e6e52c3baa4acc419608205b"
server_key_version: "encryptionkey@0"
I20260620 18:45:02.615646 19860 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 18:45:02.619362 19875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:02.620771 19860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:45:02.620958 19860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "0db553c30c5c482090e33c6691ea0cf1"
format_stamp: "Formatted at 2026-06-20 18:45:02 on dist-test-slave-k6lg"
server_key: "377e7c46fcd3879fc81706dbd2a3a120"
server_key_iv: "a611f987e6e52c3baa4acc419608205b"
server_key_version: "encryptionkey@0"
I20260620 18:45:02.621183 19860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:02.639552 19860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:02.644207 19860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:02.644515 19860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:02.663627 19860 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:36571
I20260620 18:45:02.663652 19926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:36571 every 8 connection(s)
I20260620 18:45:02.665107 19860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:45:02.667037 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 19860
I20260620 18:45:02.667271 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:45:02.668107 6022 external_mini_cluster.cc:1428] Setting key 1d54566cd6f9adb5e23d2cf1f8898b0a
I20260620 18:45:02.669940 19927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 18:45:02 dist-test-slave-k6lg krb5kdc[19847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:45:02.684376 19927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1: Bootstrap starting.
I20260620 18:45:02.687703 19927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1: Neither blocks nor log segments found. Creating new log.
I20260620 18:45:02.688762 19927 log.cc:824] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1: Log is configured to *not* fsync() on all Append() calls
I20260620 18:45:02.691856 19930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:02.670337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34868 (local address 127.5.225.190:36571)
0620 18:45:02.671146 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:02.671159 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:02.671200 (+ 41us) server_negotiation.cc:407] Connection header received
0620 18:45:02.672814 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:02.672835 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:02.673242 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:02.673577 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:02.675426 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:02.676753 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:02.678725 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:02.679424 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:02.679563 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:02.684474 (+ 4911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:02.684483 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:02.684500 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:02.684586 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:02.687659 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:02.688433 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:02.688440 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:02.688446 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:02.688619 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:02.689047 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:02.689050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:02.689052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:02.689643 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:02.690022 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:02.690430 (+ 408us) server_negotiation.cc:299] Negotiation successful
0620 18:45:02.690866 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":385,"threads_started":1}
I20260620 18:45:02.691875 19927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1: No bootstrap required, opened a new log
I20260620 18:45:02.699925 19927 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0db553c30c5c482090e33c6691ea0cf1" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36571 } }
I20260620 18:45:02.700368 19927 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:45:02.700497 19927 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0db553c30c5c482090e33c6691ea0cf1, State: Initialized, Role: FOLLOWER
I20260620 18:45:02.701169 19927 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0db553c30c5c482090e33c6691ea0cf1" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36571 } }
I20260620 18:45:02.701478 19927 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:45:02.701644 19927 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:45:02.701853 19927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:45:02.704234 19927 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0db553c30c5c482090e33c6691ea0cf1" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36571 } }
I20260620 18:45:02.704761 19927 leader_election.cc:302] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0db553c30c5c482090e33c6691ea0cf1; no voters:
I20260620 18:45:02.705381 19927 leader_election.cc:288] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:45:02.705624 19932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:45:02.706396 19932 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [term 1 LEADER]: Becoming Leader. State: Replica: 0db553c30c5c482090e33c6691ea0cf1, State: Running, Role: LEADER
I20260620 18:45:02.706957 19932 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0db553c30c5c482090e33c6691ea0cf1" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36571 } }
I20260620 18:45:02.707691 19927 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:45:02.709800 19934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0db553c30c5c482090e33c6691ea0cf1. Latest consensus state: current_term: 1 leader_uuid: "0db553c30c5c482090e33c6691ea0cf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0db553c30c5c482090e33c6691ea0cf1" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36571 } } }
I20260620 18:45:02.709928 19933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0db553c30c5c482090e33c6691ea0cf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0db553c30c5c482090e33c6691ea0cf1" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36571 } } }
I20260620 18:45:02.710244 19934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [sys.catalog]: This master's current role is: LEADER
I20260620 18:45:02.710244 19933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [sys.catalog]: This master's current role is: LEADER
I20260620 18:45:02.713954 19938 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:45:02.719295 19938 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:45:02.727990 19938 catalog_manager.cc:1347] Generated new cluster ID: ed66c37ebf06469f8c2e7e052ccb0be9
I20260620 18:45:02.728111 19938 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:45:02.737040 19938 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:45:02.738982 19938 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:45:02.748664 19938 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1: Generated new TSK 0
I20260620 18:45:02.749775 19938 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab.
Entry for principal kudu/127.5.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab.
I20260620 18:45:02.812026 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.129:0
--local_ip_for_outbound_sockets=127.5.225.129
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:36571
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:33725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:45:03.145471 19953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:03.145789 19953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:03.145975 19953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:03.153360 19953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:03.153514 19953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:03.153700 19953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.129
I20260620 18:45:03.163180 19953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:33725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:36571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.129
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:45:03.164525 19953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:03.165839 19953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:03.178845 19958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:03.178798 19959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:03.178800 19961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:03.179600 19953 server_base.cc:1034] running on GCE node
I20260620 18:45:03.180227 19953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:03.181088 19953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:03.182272 19953 hybrid_clock.cc:648] HybridClock initialized: now 1781981103182213 us; error 60 us; skew 500 ppm
I20260620 18:45:03.182543 19953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:45:03 dist-test-slave-k6lg krb5kdc[19847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981103, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:45:03.185760 19953 init.cc:377] Logged in from keytab as kudu/127.5.225.129@KRBTEST.COM (short username kudu)
I20260620 18:45:03.187327 19953 webserver.cc:460] Webserver started at http://127.5.225.129:34271/ using document root <none> and password file <none>
I20260620 18:45:03.188040 19953 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:03.188143 19953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:03.188474 19953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:45:03.191701 19953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/data/instance:
uuid: "5d9e32d518e743f5bc47654c76890cf0"
format_stamp: "Formatted at 2026-06-20 18:45:03 on dist-test-slave-k6lg"
server_key: "90a1d2725838bc673568bdb57dcc13fb"
server_key_iv: "26017eb591c73e7cbdea85d08de7f482"
server_key_version: "encryptionkey@0"
I20260620 18:45:03.192452 19953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance:
uuid: "5d9e32d518e743f5bc47654c76890cf0"
format_stamp: "Formatted at 2026-06-20 18:45:03 on dist-test-slave-k6lg"
server_key: "90a1d2725838bc673568bdb57dcc13fb"
server_key_iv: "26017eb591c73e7cbdea85d08de7f482"
server_key_version: "encryptionkey@0"
I20260620 18:45:03.197547 19953 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260620 18:45:03.201058 19968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:03.202594 19953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 18:45:03.202777 19953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/wal
uuid: "5d9e32d518e743f5bc47654c76890cf0"
format_stamp: "Formatted at 2026-06-20 18:45:03 on dist-test-slave-k6lg"
server_key: "90a1d2725838bc673568bdb57dcc13fb"
server_key_iv: "26017eb591c73e7cbdea85d08de7f482"
server_key_version: "encryptionkey@0"
I20260620 18:45:03.202992 19953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:03.219772 19953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:03.224131 19953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:03.224429 19953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:03.225544 19953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:45:03.227072 19953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:45:03.227173 19953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:03.227319 19953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:45:03.227398 19953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:03.261205 19953 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.129:37807
I20260620 18:45:03.261243 20080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.129:37807 every 8 connection(s)
I20260620 18:45:03.262614 19953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
I20260620 18:45:03.267874 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 19953
I20260620 18:45:03.268093 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance
I20260620 18:45:03.268894 6022 external_mini_cluster.cc:1428] Setting key ba8bf8587212964d1f42979f57e639d1
Jun 20 18:45:03 dist-test-slave-k6lg krb5kdc[19847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981103, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:45:03.287125 20083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:03.265299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:55129 (local address 127.5.225.190:36571)
0620 18:45:03.265789 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:03.265793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:03.266577 (+ 784us) server_negotiation.cc:407] Connection header received
0620 18:45:03.269070 (+ 2493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:03.269075 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:03.269173 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:03.269265 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:03.271156 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:03.271864 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:03.274448 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:03.274853 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:03.274952 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:03.279649 (+ 4697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:03.279657 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:03.279664 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:03.279713 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:03.283454 (+ 3741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:03.284289 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:03.284297 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:03.284304 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:03.284453 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:03.284964 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:03.284972 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:03.284978 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:03.285364 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:03.285528 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:03.286340 (+ 812us) server_negotiation.cc:299] Negotiation successful
0620 18:45:03.286625 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":251,"threads_started":1}
I20260620 18:45:03.287887 20084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:03.265656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.129:55129)
0620 18:45:03.266412 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:03.266451 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:03.268688 (+ 2237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:03.269569 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:03.269588 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:03.270255 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:03.270964 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:03.270983 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:03.272028 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:03.272035 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:03.274285 (+ 2250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:03.274304 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:03.275094 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:03.275103 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:03.275452 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:03.276250 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:03.276286 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:03.279369 (+ 3083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:45:03.283623 (+ 4254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:03.283630 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:03.283645 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:03.284143 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:03.284600 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:03.284604 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:03.284607 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:03.284826 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:03.285500 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:45:03.285506 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:45:03.285875 (+ 369us) client_negotiation.cc:769] Sending connection context
0620 18:45:03.286134 (+ 259us) client_negotiation.cc:240] Negotiation successful
0620 18:45:03.286660 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":272,"threads_started":1}
I20260620 18:45:03.289579 20081 heartbeater.cc:347] Connected to a master server at 127.5.225.190:36571
I20260620 18:45:03.289970 20081 heartbeater.cc:464] Registering TS with master...
I20260620 18:45:03.290891 20081 heartbeater.cc:511] Master 127.5.225.190:36571 requested a full tablet report, sending...
I20260620 18:45:03.293056 19892 ts_manager.cc:194] Registered new tserver with Master: 5d9e32d518e743f5bc47654c76890cf0 (127.5.225.129:37807)
WARNING: no policy specified for kudu/127.5.225.130@KRBTEST.COM; defaulting to no policy
I20260620 18:45:03.294729 19892 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.129@KRBTEST.COM'} at 127.5.225.129:55129
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab.
Entry for principal kudu/127.5.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab.
I20260620 18:45:03.323402 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.130:0
--local_ip_for_outbound_sockets=127.5.225.130
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:36571
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:33725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:45:03.646564 20087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:03.646914 20087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:03.647087 20087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:03.654428 20087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:03.654561 20087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:03.654734 20087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.130
I20260620 18:45:03.664443 20087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:33725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:36571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.130
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:45:03.665707 20087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:03.667095 20087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:03.680331 20095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:03.680349 20093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:03.680349 20092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:03.681077 20087 server_base.cc:1034] running on GCE node
I20260620 18:45:03.681708 20087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:03.682750 20087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:03.683987 20087 hybrid_clock.cc:648] HybridClock initialized: now 1781981103683916 us; error 53 us; skew 500 ppm
I20260620 18:45:03.684257 20087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:45:03 dist-test-slave-k6lg krb5kdc[19847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981103, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:45:03.687265 20087 init.cc:377] Logged in from keytab as kudu/127.5.225.130@KRBTEST.COM (short username kudu)
I20260620 18:45:03.688676 20087 webserver.cc:460] Webserver started at http://127.5.225.130:33491/ using document root <none> and password file <none>
I20260620 18:45:03.689312 20087 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:03.689419 20087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:03.689649 20087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:45:03.692569 20087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/data/instance:
uuid: "1d6dbe1a84d64c37be48f3f9acffbc62"
format_stamp: "Formatted at 2026-06-20 18:45:03 on dist-test-slave-k6lg"
server_key: "648ff48894e97290104f95b1831a6c82"
server_key_iv: "958520541ef8ad8fc68074b161ca3f18"
server_key_version: "encryptionkey@0"
I20260620 18:45:03.693260 20087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance:
uuid: "1d6dbe1a84d64c37be48f3f9acffbc62"
format_stamp: "Formatted at 2026-06-20 18:45:03 on dist-test-slave-k6lg"
server_key: "648ff48894e97290104f95b1831a6c82"
server_key_iv: "958520541ef8ad8fc68074b161ca3f18"
server_key_version: "encryptionkey@0"
I20260620 18:45:03.698305 20087 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260620 18:45:03.701573 20102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:03.703051 20087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:45:03.703224 20087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/wal
uuid: "1d6dbe1a84d64c37be48f3f9acffbc62"
format_stamp: "Formatted at 2026-06-20 18:45:03 on dist-test-slave-k6lg"
server_key: "648ff48894e97290104f95b1831a6c82"
server_key_iv: "958520541ef8ad8fc68074b161ca3f18"
server_key_version: "encryptionkey@0"
I20260620 18:45:03.703437 20087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:03.716837 20087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:03.721064 20087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:03.721351 20087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:03.722355 20087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:45:03.723624 20087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:45:03.723694 20087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:03.723853 20087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:45:03.723908 20087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:03.757759 20087 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.130:40447
I20260620 18:45:03.757799 20214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.130:40447 every 8 connection(s)
I20260620 18:45:03.759107 20087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
I20260620 18:45:03.765599 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 20087
I20260620 18:45:03.765796 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance
I20260620 18:45:03.766281 6022 external_mini_cluster.cc:1428] Setting key 4ea5dea2bec358ba3a65bf9ba93046a8
Jun 20 18:45:03 dist-test-slave-k6lg krb5kdc[19847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981103, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:45:03.779412 20083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:03.761665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:45743 (local address 127.5.225.190:36571)
0620 18:45:03.761871 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:03.761877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:03.763120 (+ 1243us) server_negotiation.cc:407] Connection header received
0620 18:45:03.765004 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:03.765009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:03.765080 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:03.765171 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:03.766914 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:03.767919 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:03.769540 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:03.770013 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:03.770149 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:03.773656 (+ 3507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:03.773661 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:03.773665 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:03.773696 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:03.776269 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:03.777024 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:03.777029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:03.777031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:03.777155 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:03.777607 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:03.777611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:03.777613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:03.777894 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:03.778051 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:03.778824 (+ 773us) server_negotiation.cc:299] Negotiation successful
0620 18:45:03.779067 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:45:03.780534 20217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:03.762256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.130:45743)
0620 18:45:03.762961 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:03.763000 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:03.764744 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:03.765383 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:03.765391 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:03.766020 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:03.766723 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:03.766742 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:03.768101 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:03.768107 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:03.769389 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:03.769401 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:03.770275 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:03.770283 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:03.770508 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:03.771216 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:03.771239 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:03.773423 (+ 2184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:45:03.776427 (+ 3004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:03.776436 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:03.776453 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:03.776885 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:03.777275 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:03.777280 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:03.777283 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:03.777480 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:03.778033 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:45:03.778042 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:45:03.778462 (+ 420us) client_negotiation.cc:769] Sending connection context
0620 18:45:03.778719 (+ 257us) client_negotiation.cc:240] Negotiation successful
0620 18:45:03.779256 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":264,"threads_started":1}
I20260620 18:45:03.782105 20215 heartbeater.cc:347] Connected to a master server at 127.5.225.190:36571
I20260620 18:45:03.782461 20215 heartbeater.cc:464] Registering TS with master...
I20260620 18:45:03.783167 20215 heartbeater.cc:511] Master 127.5.225.190:36571 requested a full tablet report, sending...
I20260620 18:45:03.784600 19892 ts_manager.cc:194] Registered new tserver with Master: 1d6dbe1a84d64c37be48f3f9acffbc62 (127.5.225.130:40447)
I20260620 18:45:03.785416 19892 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.130@KRBTEST.COM'} at 127.5.225.130:45743
WARNING: no policy specified for kudu/127.5.225.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab.
Entry for principal kudu/127.5.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab.
I20260620 18:45:03.815157 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.131:0
--local_ip_for_outbound_sockets=127.5.225.131
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:36571
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:33725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:45:04.141381 20220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:04.141665 20220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:04.141759 20220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:04.149025 20220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:04.149133 20220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:04.149245 20220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.131
I20260620 18:45:04.159262 20220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:33725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:36571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.131
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:45:04.160514 20220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:04.161741 20220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:04.174633 20228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:04.174672 20226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:04.174844 20225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:04.175557 20220 server_base.cc:1034] running on GCE node
I20260620 18:45:04.176222 20220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:04.177172 20220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:04.178376 20220 hybrid_clock.cc:648] HybridClock initialized: now 1781981104178310 us; error 63 us; skew 500 ppm
I20260620 18:45:04.178718 20220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:45:04 dist-test-slave-k6lg krb5kdc[19847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981104, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:45:04.185401 20220 init.cc:377] Logged in from keytab as kudu/127.5.225.131@KRBTEST.COM (short username kudu)
I20260620 18:45:04.187057 20220 webserver.cc:460] Webserver started at http://127.5.225.131:38251/ using document root <none> and password file <none>
I20260620 18:45:04.187777 20220 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:04.187911 20220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:04.188218 20220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:45:04.191275 20220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/data/instance:
uuid: "f0096274c1464896b4dd5597d0d98e53"
format_stamp: "Formatted at 2026-06-20 18:45:04 on dist-test-slave-k6lg"
server_key: "4bfa26af1be9e3bcdf053e01fbdccc11"
server_key_iv: "166483d8b2e52f8546ddbd68d26433ba"
server_key_version: "encryptionkey@0"
I20260620 18:45:04.192031 20220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance:
uuid: "f0096274c1464896b4dd5597d0d98e53"
format_stamp: "Formatted at 2026-06-20 18:45:04 on dist-test-slave-k6lg"
server_key: "4bfa26af1be9e3bcdf053e01fbdccc11"
server_key_iv: "166483d8b2e52f8546ddbd68d26433ba"
server_key_version: "encryptionkey@0"
I20260620 18:45:04.197410 20220 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:45:04.200752 20235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:04.202310 20220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 18:45:04.202484 20220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/wal
uuid: "f0096274c1464896b4dd5597d0d98e53"
format_stamp: "Formatted at 2026-06-20 18:45:04 on dist-test-slave-k6lg"
server_key: "4bfa26af1be9e3bcdf053e01fbdccc11"
server_key_iv: "166483d8b2e52f8546ddbd68d26433ba"
server_key_version: "encryptionkey@0"
I20260620 18:45:04.202685 20220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:04.225989 20220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:04.230600 20220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:04.230902 20220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:04.231873 20220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:45:04.233182 20220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:45:04.233294 20220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:04.233443 20220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:45:04.233525 20220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:04.267680 20220 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.131:38317
I20260620 18:45:04.267685 20347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.131:38317 every 8 connection(s)
I20260620 18:45:04.269151 20220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
I20260620 18:45:04.278533 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 20220
I20260620 18:45:04.278723 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance
I20260620 18:45:04.279187 6022 external_mini_cluster.cc:1428] Setting key 61d00c8531c3c996f52f142bd1f6e63b
Jun 20 18:45:04 dist-test-slave-k6lg krb5kdc[19847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981104, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:45:04.289678 20083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.271738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:51517 (local address 127.5.225.190:36571)
0620 18:45:04.271997 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:04.272004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:04.273048 (+ 1044us) server_negotiation.cc:407] Connection header received
0620 18:45:04.274973 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:04.274977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:04.275078 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:04.275170 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:04.276893 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.277663 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.279508 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.280093 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.280231 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.284062 (+ 3831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:04.284068 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:04.284079 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:04.284115 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:04.286640 (+ 2525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:04.287357 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:04.287363 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:04.287368 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:04.287498 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:04.287932 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:04.287936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:04.287938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:04.288207 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:04.288361 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:04.289113 (+ 752us) server_negotiation.cc:299] Negotiation successful
0620 18:45:04.289331 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260620 18:45:04.290696 20350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.272129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.131:51517)
0620 18:45:04.272873 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:04.272914 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:04.274660 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:04.275481 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:04.275492 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:04.276098 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:04.276687 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:04.276705 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.277862 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.277867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:04.279357 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:04.279369 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.280338 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.280346 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:04.280654 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.281522 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:04.281547 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:04.283847 (+ 2300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:45:04.286777 (+ 2930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:04.286786 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:04.286803 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:04.287227 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:04.287609 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:04.287615 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:04.287619 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:04.287803 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:04.288354 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:45:04.288361 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:45:04.288684 (+ 323us) client_negotiation.cc:769] Sending connection context
0620 18:45:04.288991 (+ 307us) client_negotiation.cc:240] Negotiation successful
0620 18:45:04.289527 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":276,"threads_started":1}
I20260620 18:45:04.292105 20348 heartbeater.cc:347] Connected to a master server at 127.5.225.190:36571
I20260620 18:45:04.292418 20348 heartbeater.cc:464] Registering TS with master...
I20260620 18:45:04.293198 20348 heartbeater.cc:511] Master 127.5.225.190:36571 requested a full tablet report, sending...
I20260620 18:45:04.294845 19892 ts_manager.cc:194] Registered new tserver with Master: f0096274c1464896b4dd5597d0d98e53 (127.5.225.131:38317)
I20260620 18:45:04.295651 19892 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.131@KRBTEST.COM'} at 127.5.225.131:51517
I20260620 18:45:04.298360 20081 heartbeater.cc:503] Master 127.5.225.190:36571 was elected leader, sending a full tablet report...
I20260620 18:45:04.307080 6022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:45:04.324793 20083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.311691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34882 (local address 127.5.225.190:36571)
0620 18:45:04.311919 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:04.311927 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:04.312395 (+ 468us) server_negotiation.cc:407] Connection header received
0620 18:45:04.312568 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:04.312574 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:04.312642 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:04.312742 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:04.314292 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.315039 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.316333 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.316777 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.316909 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.319338 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:04.319346 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:04.319356 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:04.319387 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:04.322059 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:04.322742 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:04.322747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:04.322751 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:04.322897 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:04.323280 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:04.323286 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:04.323290 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:04.323629 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:04.323890 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:04.324251 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 18:45:04.324470 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 18:45:04.328876 19892 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:45:04.335827 19892 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:45:04.370800 20360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.347458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.130:40447 (local address 127.0.0.1:41406)
0620 18:45:04.359858 (+ 12400us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:04.359896 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:04.360089 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:04.361007 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:04.361019 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:04.361064 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:45:04.361608 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:04.361626 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.364494 (+ 2868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.364511 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:04.366974 (+ 2463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:04.367026 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.368967 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.368977 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:04.369196 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.369911 (+ 715us) client_negotiation.cc:769] Sending connection context
0620 18:45:04.370042 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 18:45:04.370263 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12145,"mutex_wait_us":625,"thread_start_us":12004,"threads_started":1}
I20260620 18:45:04.371340 20361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.347639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41406 (local address 127.5.225.130:40447)
0620 18:45:04.359403 (+ 11764us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:04.359414 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:04.360299 (+ 885us) server_negotiation.cc:407] Connection header received
0620 18:45:04.360413 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:04.360423 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:04.360697 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:04.361879 (+ 1182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:45:04.362186 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.364263 (+ 2077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.367201 (+ 2938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.368750 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.368898 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.369101 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:04.370831 (+ 1730us) server_negotiation.cc:299] Negotiation successful
0620 18:45:04.371025 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11582,"thread_start_us":11461,"threads_started":1}
I20260620 18:45:04.374029 20362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.347817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:37807 (local address 127.0.0.1:52096)
0620 18:45:04.362647 (+ 14830us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:04.362671 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:04.362819 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:04.364256 (+ 1437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:04.364263 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:04.364291 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:45:04.364812 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:04.364824 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.367974 (+ 3150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.367980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:04.371021 (+ 3041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:04.371036 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.373239 (+ 2203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.373246 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:04.373410 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.373442 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 18:45:04.373545 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 18:45:04.373683 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14664,"thread_start_us":15107,"threads_started":1}
I20260620 18:45:04.374394 20363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.348372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52096 (local address 127.5.225.129:37807)
0620 18:45:04.363505 (+ 15133us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:04.363513 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:04.363541 (+ 28us) server_negotiation.cc:407] Connection header received
0620 18:45:04.363649 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:04.363659 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:04.363901 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:04.364106 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:45:04.365064 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.367806 (+ 2742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.371207 (+ 3401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.372721 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.372864 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.373092 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:04.373849 (+ 757us) server_negotiation.cc:299] Negotiation successful
0620 18:45:04.374021 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14958,"thread_start_us":15618,"threads_started":1}
I20260620 18:45:04.374992 20150 tablet_service.cc:1467] Processing CreateTablet for tablet d46ff713a9d346bba7cae8dfd36be2ee (DEFAULT_TABLE table=test-table [id=09e54b5422eb477da0bb0c0cd9b810d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:45:04.376587 20150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d46ff713a9d346bba7cae8dfd36be2ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:45:04.376873 20365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.364718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.131:38317 (local address 127.0.0.1:40088)
0620 18:45:04.368297 (+ 3579us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:04.368319 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:04.368467 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:04.369276 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:04.369280 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:04.369317 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:45:04.369652 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:04.369660 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.371618 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.371623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:04.373646 (+ 2023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:04.373658 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.376114 (+ 2456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.376121 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:04.376285 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.376323 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 18:45:04.376419 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 18:45:04.376551 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3446,"thread_start_us":3299,"threads_started":1}
I20260620 18:45:04.376959 20364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.360630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40088 (local address 127.5.225.131:38317)
0620 18:45:04.368006 (+ 7376us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:04.368017 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:04.368428 (+ 411us) server_negotiation.cc:407] Connection header received
0620 18:45:04.368635 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:04.368641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:04.368862 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:04.369061 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:45:04.369892 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.371433 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.374459 (+ 3026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.375903 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.376066 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.376292 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:04.376467 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 18:45:04.376622 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7194,"thread_start_us":7042,"threads_started":1}
I20260620 18:45:04.378638 20016 tablet_service.cc:1467] Processing CreateTablet for tablet d46ff713a9d346bba7cae8dfd36be2ee (DEFAULT_TABLE table=test-table [id=09e54b5422eb477da0bb0c0cd9b810d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:45:04.380065 20016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d46ff713a9d346bba7cae8dfd36be2ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:45:04.381099 20283 tablet_service.cc:1467] Processing CreateTablet for tablet d46ff713a9d346bba7cae8dfd36be2ee (DEFAULT_TABLE table=test-table [id=09e54b5422eb477da0bb0c0cd9b810d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:45:04.382395 20283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d46ff713a9d346bba7cae8dfd36be2ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:45:04.391625 20366 tablet_bootstrap.cc:492] T d46ff713a9d346bba7cae8dfd36be2ee P 1d6dbe1a84d64c37be48f3f9acffbc62: Bootstrap starting.
I20260620 18:45:04.394373 20367 tablet_bootstrap.cc:492] T d46ff713a9d346bba7cae8dfd36be2ee P f0096274c1464896b4dd5597d0d98e53: Bootstrap starting.
I20260620 18:45:04.396581 20366 tablet_bootstrap.cc:654] T d46ff713a9d346bba7cae8dfd36be2ee P 1d6dbe1a84d64c37be48f3f9acffbc62: Neither blocks nor log segments found. Creating new log.
I20260620 18:45:04.397632 20366 log.cc:824] T d46ff713a9d346bba7cae8dfd36be2ee P 1d6dbe1a84d64c37be48f3f9acffbc62: Log is configured to *not* fsync() on all Append() calls
I20260620 18:45:04.398937 20367 tablet_bootstrap.cc:654] T d46ff713a9d346bba7cae8dfd36be2ee P f0096274c1464896b4dd5597d0d98e53: Neither blocks nor log segments found. Creating new log.
I20260620 18:45:04.400247 20367 log.cc:824] T d46ff713a9d346bba7cae8dfd36be2ee P f0096274c1464896b4dd5597d0d98e53: Log is configured to *not* fsync() on all Append() calls
I20260620 18:45:04.400676 20366 tablet_bootstrap.cc:492] T d46ff713a9d346bba7cae8dfd36be2ee P 1d6dbe1a84d64c37be48f3f9acffbc62: No bootstrap required, opened a new log
I20260620 18:45:04.400939 20366 ts_tablet_manager.cc:1397] T d46ff713a9d346bba7cae8dfd36be2ee P 1d6dbe1a84d64c37be48f3f9acffbc62: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.002s
I20260620 18:45:04.402825 20367 tablet_bootstrap.cc:492] T d46ff713a9d346bba7cae8dfd36be2ee P f0096274c1464896b4dd5597d0d98e53: No bootstrap required, opened a new log
I20260620 18:45:04.403007 20368 tablet_bootstrap.cc:492] T d46ff713a9d346bba7cae8dfd36be2ee P 5d9e32d518e743f5bc47654c76890cf0: Bootstrap starting.
I20260620 18:45:04.403081 20367 ts_tablet_manager.cc:1397] T d46ff713a9d346bba7cae8dfd36be2ee P f0096274c1464896b4dd5597d0d98e53: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.001s
I20260620 18:45:04.406641 20368 tablet_bootstrap.cc:654] T d46ff713a9d346bba7cae8dfd36be2ee P 5d9e32d518e743f5bc47654c76890cf0: Neither blocks nor log segments found. Creating new log.
I20260620 18:45:04.408103 20368 log.cc:824] T d46ff713a9d346bba7cae8dfd36be2ee P 5d9e32d518e743f5bc47654c76890cf0: Log is configured to *not* fsync() on all Append() calls
I20260620 18:45:04.410780 20366 raft_consensus.cc:348] T d46ff713a9d346bba7cae8dfd36be2ee P 1d6dbe1a84d64c37be48f3f9acffbc62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dbe1a84d64c37be48f3f9acffbc62" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40447 } } peers { permanent_uuid: "5d9e32d518e743f5bc47654c76890cf0" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37807 } } peers { permanent_uuid: "f0096274c1464896b4dd5597d0d98e53" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 38317 } }
I20260620 18:45:04.411263 20366 raft_consensus.cc:374] T d46ff713a9d346bba7cae8dfd36be2ee P 1d6dbe1a84d64c37be48f3f9acffbc62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:45:04.411375 20366 raft_consensus.cc:729] T d46ff713a9d346bba7cae8dfd36be2ee P 1d6dbe1a84d64c37be48f3f9acffbc62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d6dbe1a84d64c37be48f3f9acffbc62, State: Initialized, Role: FOLLOWER
I20260620 18:45:04.411535 20368 tablet_bootstrap.cc:492] T d46ff713a9d346bba7cae8dfd36be2ee P 5d9e32d518e743f5bc47654c76890cf0: No bootstrap required, opened a new log
I20260620 18:45:04.411801 20368 ts_tablet_manager.cc:1397] T d46ff713a9d346bba7cae8dfd36be2ee P 5d9e32d518e743f5bc47654c76890cf0: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260620 18:45:04.412108 20366 consensus_queue.cc:260] T d46ff713a9d346bba7cae8dfd36be2ee P 1d6dbe1a84d64c37be48f3f9acffbc62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dbe1a84d64c37be48f3f9acffbc62" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40447 } } peers { permanent_uuid: "5d9e32d518e743f5bc47654c76890cf0" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37807 } } peers { permanent_uuid: "f0096274c1464896b4dd5597d0d98e53" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 38317 } }
I20260620 18:45:04.412145 20367 raft_consensus.cc:348] T d46ff713a9d346bba7cae8dfd36be2ee P f0096274c1464896b4dd5597d0d98e53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dbe1a84d64c37be48f3f9acffbc62" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40447 } } peers { permanent_uuid: "5d9e32d518e743f5bc47654c76890cf0" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37807 } } peers { permanent_uuid: "f0096274c1464896b4dd5597d0d98e53" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 38317 } }
I20260620 18:45:04.412494 20367 raft_consensus.cc:374] T d46ff713a9d346bba7cae8dfd36be2ee P f0096274c1464896b4dd5597d0d98e53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:45:04.412587 20367 raft_consensus.cc:729] T d46ff713a9d346bba7cae8dfd36be2ee P f0096274c1464896b4dd5597d0d98e53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0096274c1464896b4dd5597d0d98e53, State: Initialized, Role: FOLLOWER
I20260620 18:45:04.413141 20367 consensus_queue.cc:260] T d46ff713a9d346bba7cae8dfd36be2ee P f0096274c1464896b4dd5597d0d98e53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dbe1a84d64c37be48f3f9acffbc62" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40447 } } peers { permanent_uuid: "5d9e32d518e743f5bc47654c76890cf0" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37807 } } peers { permanent_uuid: "f0096274c1464896b4dd5597d0d98e53" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 38317 } }
I20260620 18:45:04.414264 20348 heartbeater.cc:503] Master 127.5.225.190:36571 was elected leader, sending a full tablet report...
I20260620 18:45:04.416517 20367 ts_tablet_manager.cc:1428] T d46ff713a9d346bba7cae8dfd36be2ee P f0096274c1464896b4dd5597d0d98e53: Time spent starting tablet: real 0.013s user 0.009s sys 0.002s
I20260620 18:45:04.417462 20215 heartbeater.cc:503] Master 127.5.225.190:36571 was elected leader, sending a full tablet report...
I20260620 18:45:04.419011 20366 ts_tablet_manager.cc:1428] T d46ff713a9d346bba7cae8dfd36be2ee P 1d6dbe1a84d64c37be48f3f9acffbc62: Time spent starting tablet: real 0.018s user 0.012s sys 0.006s
I20260620 18:45:04.422857 20368 raft_consensus.cc:348] T d46ff713a9d346bba7cae8dfd36be2ee P 5d9e32d518e743f5bc47654c76890cf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dbe1a84d64c37be48f3f9acffbc62" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40447 } } peers { permanent_uuid: "5d9e32d518e743f5bc47654c76890cf0" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37807 } } peers { permanent_uuid: "f0096274c1464896b4dd5597d0d98e53" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 38317 } }
I20260620 18:45:04.423168 20368 raft_consensus.cc:374] T d46ff713a9d346bba7cae8dfd36be2ee P 5d9e32d518e743f5bc47654c76890cf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:45:04.423237 20368 raft_consensus.cc:729] T d46ff713a9d346bba7cae8dfd36be2ee P 5d9e32d518e743f5bc47654c76890cf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d9e32d518e743f5bc47654c76890cf0, State: Initialized, Role: FOLLOWER
I20260620 18:45:04.423688 20368 consensus_queue.cc:260] T d46ff713a9d346bba7cae8dfd36be2ee P 5d9e32d518e743f5bc47654c76890cf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dbe1a84d64c37be48f3f9acffbc62" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40447 } } peers { permanent_uuid: "5d9e32d518e743f5bc47654c76890cf0" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37807 } } peers { permanent_uuid: "f0096274c1464896b4dd5597d0d98e53" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 38317 } }
I20260620 18:45:04.425173 20368 ts_tablet_manager.cc:1428] T d46ff713a9d346bba7cae8dfd36be2ee P 5d9e32d518e743f5bc47654c76890cf0: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
W20260620 18:45:04.511077 20216 tablet.cc:2369] T d46ff713a9d346bba7cae8dfd36be2ee P 1d6dbe1a84d64c37be48f3f9acffbc62: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:45:04.515012 20082 tablet.cc:2369] T d46ff713a9d346bba7cae8dfd36be2ee P 5d9e32d518e743f5bc47654c76890cf0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:45:04.520695 20349 tablet.cc:2369] T d46ff713a9d346bba7cae8dfd36be2ee P f0096274c1464896b4dd5597d0d98e53: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:45:04.584401 20374 raft_consensus.cc:482] T d46ff713a9d346bba7cae8dfd36be2ee P 5d9e32d518e743f5bc47654c76890cf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:45:04.584718 20374 raft_consensus.cc:504] T d46ff713a9d346bba7cae8dfd36be2ee P 5d9e32d518e743f5bc47654c76890cf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dbe1a84d64c37be48f3f9acffbc62" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40447 } } peers { permanent_uuid: "5d9e32d518e743f5bc47654c76890cf0" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37807 } } peers { permanent_uuid: "f0096274c1464896b4dd5597d0d98e53" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 38317 } }
I20260620 18:45:04.586578 20374 leader_election.cc:288] T d46ff713a9d346bba7cae8dfd36be2ee P 5d9e32d518e743f5bc47654c76890cf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d6dbe1a84d64c37be48f3f9acffbc62 (127.5.225.130:40447), f0096274c1464896b4dd5597d0d98e53 (127.5.225.131:38317)
I20260620 18:45:04.594223 20361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.586961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:40357 (local address 127.5.225.130:40447)
0620 18:45:04.587231 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:04.587238 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:04.587669 (+ 431us) server_negotiation.cc:407] Connection header received
0620 18:45:04.587925 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:04.587933 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:04.588016 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:04.588086 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:45:04.588963 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.589929 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.591953 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.593147 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.593269 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.593325 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:04.593750 (+ 425us) server_negotiation.cc:299] Negotiation successful
0620 18:45:04.593939 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 18:45:04.594281 20375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.586789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.130:40447 (local address 127.5.225.129:40357)
0620 18:45:04.587519 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:04.587548 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:04.587733 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:04.588298 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:04.588301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:04.588325 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:45:04.588785 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:04.588799 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.590069 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.590075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:04.591794 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:04.591806 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.593362 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.593370 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:04.593577 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.593616 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 18:45:04.593713 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 18:45:04.593826 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"spinlock_wait_cycles":36992,"thread_start_us":370,"threads_started":1}
I20260620 18:45:04.594442 20364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.587813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:51973 (local address 127.5.225.131:38317)
0620 18:45:04.588140 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:04.588145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:04.588410 (+ 265us) server_negotiation.cc:407] Connection header received
0620 18:45:04.588728 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:04.588735 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:04.588813 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:04.588901 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:45:04.589487 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.590698 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.592585 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.593461 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.593613 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.593689 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:04.594040 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 18:45:04.594190 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260620 18:45:04.594766 20376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.587619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.131:38317 (local address 127.5.225.129:51973)
0620 18:45:04.588287 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:04.588301 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:04.588561 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:04.588930 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:04.588936 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:04.588960 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:45:04.589329 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:04.589342 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.590843 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.590847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:04.592441 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:04.592456 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.593656 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.593661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:04.593854 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.593886 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 18:45:04.593981 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 18:45:04.594110 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"spinlock_wait_cycles":25856,"thread_start_us":317,"threads_started":1}
I20260620 18:45:04.595497 20170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d46ff713a9d346bba7cae8dfd36be2ee" candidate_uuid: "5d9e32d518e743f5bc47654c76890cf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d6dbe1a84d64c37be48f3f9acffbc62" is_pre_election: true
I20260620 18:45:04.595714 20303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d46ff713a9d346bba7cae8dfd36be2ee" candidate_uuid: "5d9e32d518e743f5bc47654c76890cf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0096274c1464896b4dd5597d0d98e53" is_pre_election: true
I20260620 18:45:04.596107 20170 raft_consensus.cc:2454] T d46ff713a9d346bba7cae8dfd36be2ee P 1d6dbe1a84d64c37be48f3f9acffbc62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d9e32d518e743f5bc47654c76890cf0 in term 0.
I20260620 18:45:04.596212 20303 raft_consensus.cc:2454] T d46ff713a9d346bba7cae8dfd36be2ee P f0096274c1464896b4dd5597d0d98e53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d9e32d518e743f5bc47654c76890cf0 in term 0.
I20260620 18:45:04.596812 19971 leader_election.cc:302] T d46ff713a9d346bba7cae8dfd36be2ee P 5d9e32d518e743f5bc47654c76890cf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d9e32d518e743f5bc47654c76890cf0, f0096274c1464896b4dd5597d0d98e53; no voters:
I20260620 18:45:04.597162 20374 raft_consensus.cc:2781] T d46ff713a9d346bba7cae8dfd36be2ee P 5d9e32d518e743f5bc47654c76890cf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:45:04.597334 20374 raft_consensus.cc:482] T d46ff713a9d346bba7cae8dfd36be2ee P 5d9e32d518e743f5bc47654c76890cf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:45:04.597462 20374 raft_consensus.cc:3037] T d46ff713a9d346bba7cae8dfd36be2ee P 5d9e32d518e743f5bc47654c76890cf0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:45:04.599745 20374 raft_consensus.cc:504] T d46ff713a9d346bba7cae8dfd36be2ee P 5d9e32d518e743f5bc47654c76890cf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dbe1a84d64c37be48f3f9acffbc62" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40447 } } peers { permanent_uuid: "5d9e32d518e743f5bc47654c76890cf0" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37807 } } peers { permanent_uuid: "f0096274c1464896b4dd5597d0d98e53" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 38317 } }
I20260620 18:45:04.600387 20374 leader_election.cc:288] T d46ff713a9d346bba7cae8dfd36be2ee P 5d9e32d518e743f5bc47654c76890cf0 [CANDIDATE]: Term 1 election: Requested vote from peers 1d6dbe1a84d64c37be48f3f9acffbc62 (127.5.225.130:40447), f0096274c1464896b4dd5597d0d98e53 (127.5.225.131:38317)
I20260620 18:45:04.600898 20170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d46ff713a9d346bba7cae8dfd36be2ee" candidate_uuid: "5d9e32d518e743f5bc47654c76890cf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d6dbe1a84d64c37be48f3f9acffbc62"
I20260620 18:45:04.601145 20170 raft_consensus.cc:3037] T d46ff713a9d346bba7cae8dfd36be2ee P 1d6dbe1a84d64c37be48f3f9acffbc62 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:45:04.601265 20303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d46ff713a9d346bba7cae8dfd36be2ee" candidate_uuid: "5d9e32d518e743f5bc47654c76890cf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0096274c1464896b4dd5597d0d98e53"
I20260620 18:45:04.601516 20303 raft_consensus.cc:3037] T d46ff713a9d346bba7cae8dfd36be2ee P f0096274c1464896b4dd5597d0d98e53 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:45:04.603538 20170 raft_consensus.cc:2454] T d46ff713a9d346bba7cae8dfd36be2ee P 1d6dbe1a84d64c37be48f3f9acffbc62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d9e32d518e743f5bc47654c76890cf0 in term 1.
I20260620 18:45:04.603883 20303 raft_consensus.cc:2454] T d46ff713a9d346bba7cae8dfd36be2ee P f0096274c1464896b4dd5597d0d98e53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d9e32d518e743f5bc47654c76890cf0 in term 1.
I20260620 18:45:04.603914 19971 leader_election.cc:302] T d46ff713a9d346bba7cae8dfd36be2ee P 5d9e32d518e743f5bc47654c76890cf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d6dbe1a84d64c37be48f3f9acffbc62, 5d9e32d518e743f5bc47654c76890cf0; no voters:
I20260620 18:45:04.604252 20374 raft_consensus.cc:2781] T d46ff713a9d346bba7cae8dfd36be2ee P 5d9e32d518e743f5bc47654c76890cf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:45:04.604701 20374 raft_consensus.cc:686] T d46ff713a9d346bba7cae8dfd36be2ee P 5d9e32d518e743f5bc47654c76890cf0 [term 1 LEADER]: Becoming Leader. State: Replica: 5d9e32d518e743f5bc47654c76890cf0, State: Running, Role: LEADER
I20260620 18:45:04.605294 20374 consensus_queue.cc:237] T d46ff713a9d346bba7cae8dfd36be2ee P 5d9e32d518e743f5bc47654c76890cf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dbe1a84d64c37be48f3f9acffbc62" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40447 } } peers { permanent_uuid: "5d9e32d518e743f5bc47654c76890cf0" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37807 } } peers { permanent_uuid: "f0096274c1464896b4dd5597d0d98e53" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 38317 } }
I20260620 18:45:04.610509 19891 catalog_manager.cc:5495] T d46ff713a9d346bba7cae8dfd36be2ee P 5d9e32d518e743f5bc47654c76890cf0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d9e32d518e743f5bc47654c76890cf0 (127.5.225.129). New cstate: current_term: 1 leader_uuid: "5d9e32d518e743f5bc47654c76890cf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dbe1a84d64c37be48f3f9acffbc62" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d9e32d518e743f5bc47654c76890cf0" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0096274c1464896b4dd5597d0d98e53" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 38317 } health_report { overall_health: UNKNOWN } } }
I20260620 18:45:04.672190 20363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.666695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52102 (local address 127.5.225.129:37807)
0620 18:45:04.666926 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:04.666932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:04.666951 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:45:04.667187 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:04.667191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:04.667271 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:04.667428 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:45:04.667804 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.668524 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.669945 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.670426 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.670571 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.671085 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:45:04.671602 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:45:04.671768 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:04.671825 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 18:45:04.671910 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 18:45:04.680575 20170 raft_consensus.cc:1261] T d46ff713a9d346bba7cae8dfd36be2ee P 1d6dbe1a84d64c37be48f3f9acffbc62 [term 1 FOLLOWER]: Refusing update from remote peer 5d9e32d518e743f5bc47654c76890cf0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:45:04.680589 20303 raft_consensus.cc:1261] T d46ff713a9d346bba7cae8dfd36be2ee P f0096274c1464896b4dd5597d0d98e53 [term 1 FOLLOWER]: Refusing update from remote peer 5d9e32d518e743f5bc47654c76890cf0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:45:04.681560 20377 consensus_queue.cc:1035] T d46ff713a9d346bba7cae8dfd36be2ee P 5d9e32d518e743f5bc47654c76890cf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0096274c1464896b4dd5597d0d98e53" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 38317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:45:04.681928 20374 consensus_queue.cc:1035] T d46ff713a9d346bba7cae8dfd36be2ee P 5d9e32d518e743f5bc47654c76890cf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d6dbe1a84d64c37be48f3f9acffbc62" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:45:04.694255 20382 mvcc.cc:205] Tried to move back new op lower bound from 7298994604761153536 to 7298994604467470336. Current Snapshot: MvccSnapshot[applied={T|T < 7298994604761153536}]
I20260620 18:45:04.705636 19892 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:45:04.705935 19892 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:45:04.710126 19892 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1: Sending DeleteTablet for 3 replicas of tablet d46ff713a9d346bba7cae8dfd36be2ee
I20260620 18:45:04.711215 20150 tablet_service.cc:1514] Processing DeleteTablet for tablet d46ff713a9d346bba7cae8dfd36be2ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:45:04 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:41406
I20260620 18:45:04.711265 20016 tablet_service.cc:1514] Processing DeleteTablet for tablet d46ff713a9d346bba7cae8dfd36be2ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:45:04 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:52096
I20260620 18:45:04.712132 20390 tablet_replica.cc:330] T d46ff713a9d346bba7cae8dfd36be2ee P 1d6dbe1a84d64c37be48f3f9acffbc62: stopping tablet replica
I20260620 18:45:04.712488 20390 raft_consensus.cc:2229] T d46ff713a9d346bba7cae8dfd36be2ee P 1d6dbe1a84d64c37be48f3f9acffbc62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:45:04.712895 20390 raft_consensus.cc:2258] T d46ff713a9d346bba7cae8dfd36be2ee P 1d6dbe1a84d64c37be48f3f9acffbc62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:45:04.714704 20283 tablet_service.cc:1514] Processing DeleteTablet for tablet d46ff713a9d346bba7cae8dfd36be2ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:45:04 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:40088
I20260620 18:45:04.715092 20390 ts_tablet_manager.cc:1905] T d46ff713a9d346bba7cae8dfd36be2ee P 1d6dbe1a84d64c37be48f3f9acffbc62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:45:04.715636 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 19953
I20260620 18:45:04.723212 20390 ts_tablet_manager.cc:1918] T d46ff713a9d346bba7cae8dfd36be2ee P 1d6dbe1a84d64c37be48f3f9acffbc62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:45:04.723384 20390 log.cc:1196] T d46ff713a9d346bba7cae8dfd36be2ee P 1d6dbe1a84d64c37be48f3f9acffbc62: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/wal/wals/d46ff713a9d346bba7cae8dfd36be2ee
I20260620 18:45:04.723949 20390 ts_tablet_manager.cc:1939] T d46ff713a9d346bba7cae8dfd36be2ee P 1d6dbe1a84d64c37be48f3f9acffbc62: Deleting consensus metadata
I20260620 18:45:04.724604 20391 tablet_replica.cc:330] T d46ff713a9d346bba7cae8dfd36be2ee P 5d9e32d518e743f5bc47654c76890cf0: stopping tablet replica
I20260620 18:45:04.725090 20391 raft_consensus.cc:2229] T d46ff713a9d346bba7cae8dfd36be2ee P 5d9e32d518e743f5bc47654c76890cf0 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:45:04.725785 20391 raft_consensus.cc:2258] T d46ff713a9d346bba7cae8dfd36be2ee P 5d9e32d518e743f5bc47654c76890cf0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:45:04.726059 19878 catalog_manager.cc:4841] TS 1d6dbe1a84d64c37be48f3f9acffbc62 (127.5.225.130:40447): tablet d46ff713a9d346bba7cae8dfd36be2ee (table test-table [id=09e54b5422eb477da0bb0c0cd9b810d3]) successfully deleted
I20260620 18:45:04.731405 20392 tablet_replica.cc:330] T d46ff713a9d346bba7cae8dfd36be2ee P f0096274c1464896b4dd5597d0d98e53: stopping tablet replica
I20260620 18:45:04.731830 20392 raft_consensus.cc:2229] T d46ff713a9d346bba7cae8dfd36be2ee P f0096274c1464896b4dd5597d0d98e53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:45:04.732074 20391 ts_tablet_manager.cc:1905] T d46ff713a9d346bba7cae8dfd36be2ee P 5d9e32d518e743f5bc47654c76890cf0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:45:04.732322 20392 raft_consensus.cc:2258] T d46ff713a9d346bba7cae8dfd36be2ee P f0096274c1464896b4dd5597d0d98e53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:45:04.735666 20392 ts_tablet_manager.cc:1905] T d46ff713a9d346bba7cae8dfd36be2ee P f0096274c1464896b4dd5597d0d98e53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:45:04.740190 20391 ts_tablet_manager.cc:1918] T d46ff713a9d346bba7cae8dfd36be2ee P 5d9e32d518e743f5bc47654c76890cf0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:45:04.740365 20391 log.cc:1196] T d46ff713a9d346bba7cae8dfd36be2ee P 5d9e32d518e743f5bc47654c76890cf0: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/wal/wals/d46ff713a9d346bba7cae8dfd36be2ee
I20260620 18:45:04.740911 20391 ts_tablet_manager.cc:1939] T d46ff713a9d346bba7cae8dfd36be2ee P 5d9e32d518e743f5bc47654c76890cf0: Deleting consensus metadata
I20260620 18:45:04.742671 19879 catalog_manager.cc:4841] TS 5d9e32d518e743f5bc47654c76890cf0 (127.5.225.129:37807): tablet d46ff713a9d346bba7cae8dfd36be2ee (table test-table [id=09e54b5422eb477da0bb0c0cd9b810d3]) successfully deleted
Jun 20 18:45:04 dist-test-slave-k6lg krb5kdc[19847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.129@KRBTEST.COM
I20260620 18:45:04.743110 20392 ts_tablet_manager.cc:1918] T d46ff713a9d346bba7cae8dfd36be2ee P f0096274c1464896b4dd5597d0d98e53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:45:04.743287 20392 log.cc:1196] T d46ff713a9d346bba7cae8dfd36be2ee P f0096274c1464896b4dd5597d0d98e53: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/wal/wals/d46ff713a9d346bba7cae8dfd36be2ee
I20260620 18:45:04.743695 20392 ts_tablet_manager.cc:1939] T d46ff713a9d346bba7cae8dfd36be2ee P f0096274c1464896b4dd5597d0d98e53: Deleting consensus metadata
I20260620 18:45:04.745731 19878 catalog_manager.cc:4841] TS f0096274c1464896b4dd5597d0d98e53 (127.5.225.131:38317): tablet d46ff713a9d346bba7cae8dfd36be2ee (table test-table [id=09e54b5422eb477da0bb0c0cd9b810d3]) successfully deleted
I20260620 18:45:04.750811 20363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.716868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52116 (local address 127.5.225.129:37807)
0620 18:45:04.717040 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:04.717051 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:04.730775 (+ 13724us) server_negotiation.cc:407] Connection header received
0620 18:45:04.731006 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:04.731014 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:04.731107 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:04.731202 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:04.734125 (+ 2923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.735451 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.737438 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.738227 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.738340 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.744238 (+ 5898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:04.744255 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:04.744273 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:04.744312 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:04.747819 (+ 3507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:04.748661 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:04.748669 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:04.748671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:04.748835 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:04.749246 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:04.749249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:04.749251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:04.749604 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:04.749837 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:04.750285 (+ 448us) server_negotiation.cc:299] Negotiation successful
0620 18:45:04.750463 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 18:45:04.751619 20076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:04.879426 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 19953
I20260620 18:45:04.898116 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 20087
Jun 20 18:45:04 dist-test-slave-k6lg krb5kdc[19847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.130@KRBTEST.COM
I20260620 18:45:04.914441 20361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.899083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41410 (local address 127.5.225.130:40447)
0620 18:45:04.899268 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:04.899276 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:04.899298 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:45:04.899516 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:04.899522 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:04.899594 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:04.899701 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:04.901163 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.901930 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.903320 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.903719 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.903883 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.908682 (+ 4799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:04.908730 (+ 48us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:04.908755 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:04.908802 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:04.911495 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:04.912176 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:04.912185 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:04.912189 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:04.912317 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:04.912732 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:04.912738 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:04.912742 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:04.913241 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:04.913565 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:04.913843 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 18:45:04.914057 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:45:04.915158 20210 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:05.005705 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 20087
I20260620 18:45:05.023751 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 20220
Jun 20 18:45:05 dist-test-slave-k6lg krb5kdc[19847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.131@KRBTEST.COM
I20260620 18:45:05.039646 20364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:05.024568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40092 (local address 127.5.225.131:38317)
0620 18:45:05.024781 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:05.024788 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:05.024809 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:45:05.024997 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:05.025009 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:05.025103 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:05.025213 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:05.026719 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:05.027477 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:05.028788 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:05.029181 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:05.029314 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:05.033875 (+ 4561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:05.033890 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:05.033907 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:05.033948 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:05.036637 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:05.037321 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:05.037328 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:05.037331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:05.037485 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:05.037891 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:05.037896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:05.037905 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:05.038445 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:05.038747 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:05.039046 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 18:45:05.039271 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:45:05.040326 20343 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:05.129621 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 20220
I20260620 18:45:05.146368 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 19860
I20260620 18:45:05.147583 19922 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:05.260546 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 19860
I20260620 18:45:05.282567 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:36571
--webserver_interface=127.5.225.190
--webserver_port=44363
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:33725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:36571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:45:05.613130 20398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:05.613420 20398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:05.613595 20398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:05.620893 20398 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:45:05.621024 20398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:05.621125 20398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:05.621217 20398 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:45:05.621279 20398 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:45:05.631801 20398 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:33725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:36571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:36571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=44363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:45:05.633201 20398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:05.634660 20398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:05.658456 20407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:05.658418 20405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:05.658432 20404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:05.664232 20398 server_base.cc:1034] running on GCE node
I20260620 18:45:05.665244 20398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:05.667114 20398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:05.668367 20398 hybrid_clock.cc:648] HybridClock initialized: now 1781981105668324 us; error 58 us; skew 500 ppm
I20260620 18:45:05.668804 20398 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:45:05 dist-test-slave-k6lg krb5kdc[19847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981105, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:45:05.672495 20398 init.cc:377] Logged in from keytab as kudu/127.5.225.190@KRBTEST.COM (short username kudu)
I20260620 18:45:05.674372 20398 webserver.cc:460] Webserver started at http://127.5.225.190:44363/ using document root <none> and password file <none>
I20260620 18:45:05.675190 20398 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:05.675362 20398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:05.681689 20398 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 18:45:05.684859 20414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:05.686404 20398 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:45:05.686607 20398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "0db553c30c5c482090e33c6691ea0cf1"
format_stamp: "Formatted at 2026-06-20 18:45:02 on dist-test-slave-k6lg"
server_key: "377e7c46fcd3879fc81706dbd2a3a120"
server_key_iv: "a611f987e6e52c3baa4acc419608205b"
server_key_version: "encryptionkey@0"
I20260620 18:45:05.687740 20398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:05.704603 20398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:05.709379 20398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:05.709697 20398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:05.729681 20398 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:36571
I20260620 18:45:05.729689 20465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:36571 every 8 connection(s)
I20260620 18:45:05.731359 20398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:45:05.735154 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 20398
I20260620 18:45:05.736160 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.129:37807
--local_ip_for_outbound_sockets=127.5.225.129
--tserver_master_addrs=127.5.225.190:36571
--webserver_port=34271
--webserver_interface=127.5.225.129
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:33725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:45:05.740513 20466 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:45:05.745666 20466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1: Bootstrap starting.
I20260620 18:45:05.766096 20466 log.cc:824] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1: Log is configured to *not* fsync() on all Append() calls
I20260620 18:45:05.778213 20466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:45:05.778748 20466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1: Bootstrap complete.
I20260620 18:45:05.788000 20466 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0db553c30c5c482090e33c6691ea0cf1" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36571 } }
I20260620 18:45:05.788851 20466 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0db553c30c5c482090e33c6691ea0cf1, State: Initialized, Role: FOLLOWER
I20260620 18:45:05.789565 20466 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0db553c30c5c482090e33c6691ea0cf1" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36571 } }
I20260620 18:45:05.789822 20466 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:45:05.789945 20466 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:45:05.790129 20466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:45:05.793983 20466 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0db553c30c5c482090e33c6691ea0cf1" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36571 } }
I20260620 18:45:05.794623 20466 leader_election.cc:302] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0db553c30c5c482090e33c6691ea0cf1; no voters:
I20260620 18:45:05.795269 20466 leader_election.cc:288] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 18:45:05.795408 20470 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 18:45:05.797696 20466 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:45:05.802495 20470 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [term 2 LEADER]: Becoming Leader. State: Replica: 0db553c30c5c482090e33c6691ea0cf1, State: Running, Role: LEADER
I20260620 18:45:05.803172 20470 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0db553c30c5c482090e33c6691ea0cf1" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36571 } }
I20260620 18:45:05.807657 20471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0db553c30c5c482090e33c6691ea0cf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0db553c30c5c482090e33c6691ea0cf1" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36571 } } }
I20260620 18:45:05.808118 20471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [sys.catalog]: This master's current role is: LEADER
I20260620 18:45:05.808753 20472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0db553c30c5c482090e33c6691ea0cf1. Latest consensus state: current_term: 2 leader_uuid: "0db553c30c5c482090e33c6691ea0cf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0db553c30c5c482090e33c6691ea0cf1" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36571 } } }
I20260620 18:45:05.809064 20472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [sys.catalog]: This master's current role is: LEADER
I20260620 18:45:05.812734 20485 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:45:05.818960 20485 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:45:05.819999 20485 catalog_manager.cc:1259] Loaded cluster ID: ed66c37ebf06469f8c2e7e052ccb0be9
I20260620 18:45:05.820111 20485 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:45:05.822358 20485 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 18:45:05.833989 20485 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 18:45:05.841627 20485 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:45:05.843111 20485 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1: Loaded TSK: 0
I20260620 18:45:05.844237 20485 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 18:45:06.117216 20468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:06.117552 20468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:06.117712 20468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:06.125820 20468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:06.125953 20468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:06.126123 20468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.129
I20260620 18:45:06.136181 20468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:33725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.129:37807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.225.129
--webserver_port=34271
--tserver_master_addrs=127.5.225.190:36571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.129
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:45:06.137526 20468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:06.138947 20468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:06.154377 20495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:06.154506 20497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:06.155601 20468 server_base.cc:1034] running on GCE node
W20260620 18:45:06.155931 20494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:06.156672 20468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:06.157776 20468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:06.159013 20468 hybrid_clock.cc:648] HybridClock initialized: now 1781981106158966 us; error 33 us; skew 500 ppm
I20260620 18:45:06.159368 20468 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:45:06 dist-test-slave-k6lg krb5kdc[19847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981106, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:45:06.162730 20468 init.cc:377] Logged in from keytab as kudu/127.5.225.129@KRBTEST.COM (short username kudu)
I20260620 18:45:06.164296 20468 webserver.cc:460] Webserver started at http://127.5.225.129:34271/ using document root <none> and password file <none>
I20260620 18:45:06.165020 20468 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:06.165179 20468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:06.171339 20468 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 18:45:06.174292 20504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:06.175755 20468 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:45:06.175984 20468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/wal
uuid: "5d9e32d518e743f5bc47654c76890cf0"
format_stamp: "Formatted at 2026-06-20 18:45:03 on dist-test-slave-k6lg"
server_key: "90a1d2725838bc673568bdb57dcc13fb"
server_key_iv: "26017eb591c73e7cbdea85d08de7f482"
server_key_version: "encryptionkey@0"
I20260620 18:45:06.176995 20468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:06.204973 20468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:06.209650 20468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:06.209941 20468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:06.211058 20468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:45:06.212441 20468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:45:06.212563 20468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:06.212700 20468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:45:06.212787 20468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:06.245679 20468 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.129:37807
I20260620 18:45:06.245766 20616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.129:37807 every 8 connection(s)
I20260620 18:45:06.247210 20468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
I20260620 18:45:06.253252 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 20468
I20260620 18:45:06.253996 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.130:40447
--local_ip_for_outbound_sockets=127.5.225.130
--tserver_master_addrs=127.5.225.190:36571
--webserver_port=33491
--webserver_interface=127.5.225.130
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:33725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:45:06 dist-test-slave-k6lg krb5kdc[19847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981106, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:45:06.272423 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:06.250285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.129:56483)
0620 18:45:06.251094 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:06.251140 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:06.252864 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:06.254600 (+ 1736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:06.254607 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:06.255152 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:06.255687 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:06.255704 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:06.258109 (+ 2405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:06.258113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:06.259476 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:06.259488 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:06.260975 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:06.260980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:06.261322 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:06.262072 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:06.262110 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:06.264491 (+ 2381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:45:06.267964 (+ 3473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:06.267973 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:06.267989 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:06.268417 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:06.268864 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:06.268869 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:06.268873 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:06.269062 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:06.269838 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:45:06.269847 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:45:06.270238 (+ 391us) client_negotiation.cc:769] Sending connection context
0620 18:45:06.270510 (+ 272us) client_negotiation.cc:240] Negotiation successful
0620 18:45:06.271072 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":333,"threads_started":1}
I20260620 18:45:06.272429 20619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:06.250032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:56483 (local address 127.5.225.190:36571)
0620 18:45:06.250661 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:06.250673 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:06.251280 (+ 607us) server_negotiation.cc:407] Connection header received
0620 18:45:06.253524 (+ 2244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:06.253545 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:06.254029 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:06.254451 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:06.255921 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:06.257924 (+ 2003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:06.259625 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:06.260751 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:06.260952 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:06.264773 (+ 3821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:06.264782 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:06.264797 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:06.264834 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:06.267793 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:06.268570 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:06.268578 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:06.268583 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:06.268741 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:06.269196 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:06.269200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:06.269203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:06.269667 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:06.269893 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:06.270738 (+ 845us) server_negotiation.cc:299] Negotiation successful
0620 18:45:06.271118 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":301,"threads_started":1}
I20260620 18:45:06.275672 20617 heartbeater.cc:347] Connected to a master server at 127.5.225.190:36571
I20260620 18:45:06.276050 20617 heartbeater.cc:464] Registering TS with master...
I20260620 18:45:06.277025 20617 heartbeater.cc:511] Master 127.5.225.190:36571 requested a full tablet report, sending...
I20260620 18:45:06.279443 20431 ts_manager.cc:194] Registered new tserver with Master: 5d9e32d518e743f5bc47654c76890cf0 (127.5.225.129:37807)
I20260620 18:45:06.281049 20431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.129@KRBTEST.COM'} at 127.5.225.129:56483
W20260620 18:45:06.591388 20621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:06.591722 20621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:06.591861 20621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:06.599509 20621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:06.599618 20621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:06.599805 20621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.130
I20260620 18:45:06.610224 20621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:33725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.130:40447
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.225.130
--webserver_port=33491
--tserver_master_addrs=127.5.225.190:36571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.130
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:45:06.611563 20621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:06.612888 20621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:06.625959 20626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:06.625993 20627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:06.626125 20629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:06.626776 20621 server_base.cc:1034] running on GCE node
I20260620 18:45:06.627462 20621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:06.628400 20621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:06.629604 20621 hybrid_clock.cc:648] HybridClock initialized: now 1781981106629544 us; error 54 us; skew 500 ppm
I20260620 18:45:06.629937 20621 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:45:06 dist-test-slave-k6lg krb5kdc[19847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981106, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:45:06.633260 20621 init.cc:377] Logged in from keytab as kudu/127.5.225.130@KRBTEST.COM (short username kudu)
I20260620 18:45:06.634865 20621 webserver.cc:460] Webserver started at http://127.5.225.130:33491/ using document root <none> and password file <none>
I20260620 18:45:06.635572 20621 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:06.635733 20621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:06.641499 20621 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 18:45:06.644466 20636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:06.645838 20621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:45:06.646054 20621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/wal
uuid: "1d6dbe1a84d64c37be48f3f9acffbc62"
format_stamp: "Formatted at 2026-06-20 18:45:03 on dist-test-slave-k6lg"
server_key: "648ff48894e97290104f95b1831a6c82"
server_key_iv: "958520541ef8ad8fc68074b161ca3f18"
server_key_version: "encryptionkey@0"
I20260620 18:45:06.647140 20621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:06.670262 20621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:06.674675 20621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:06.674955 20621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:06.675988 20621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:45:06.677323 20621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:45:06.677445 20621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:06.677580 20621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:45:06.677666 20621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:06.711709 20621 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.130:40447
I20260620 18:45:06.711796 20748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.130:40447 every 8 connection(s)
I20260620 18:45:06.713199 20621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
I20260620 18:45:06.719266 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 20621
I20260620 18:45:06.720096 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.131:38317
--local_ip_for_outbound_sockets=127.5.225.131
--tserver_master_addrs=127.5.225.190:36571
--webserver_port=38251
--webserver_interface=127.5.225.131
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:33725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:45:06 dist-test-slave-k6lg krb5kdc[19847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981106, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:45:06.737557 20619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:06.716138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:57545 (local address 127.5.225.190:36571)
0620 18:45:06.716404 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:06.716411 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:06.717672 (+ 1261us) server_negotiation.cc:407] Connection header received
0620 18:45:06.719726 (+ 2054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:06.719733 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:06.719838 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:06.719964 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:06.721578 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:06.722884 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:06.724465 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:06.725335 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:06.725519 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:06.729952 (+ 4433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:06.729960 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:06.729966 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:06.730010 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:06.733746 (+ 3736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:06.734714 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:06.734721 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:06.734725 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:06.734884 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:06.735390 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:06.735395 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:06.735399 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:06.735791 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:06.735942 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:06.736887 (+ 945us) server_negotiation.cc:299] Negotiation successful
0620 18:45:06.737175 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 18:45:06.738933 20751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:06.716767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.130:57545)
0620 18:45:06.717503 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:06.717548 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:06.719354 (+ 1806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:06.720183 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:06.720215 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:06.720796 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:06.721388 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:06.721409 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:06.723052 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:06.723055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:06.724300 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:06.724311 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:06.725618 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:06.725625 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:06.725931 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:06.726699 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:06.726727 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:06.729701 (+ 2974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:45:06.733910 (+ 4209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:06.733922 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:06.733941 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:06.734586 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:06.735014 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:06.735021 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:06.735028 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:06.735264 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:06.735944 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:45:06.735956 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:45:06.736410 (+ 454us) client_negotiation.cc:769] Sending connection context
0620 18:45:06.736725 (+ 315us) client_negotiation.cc:240] Negotiation successful
0620 18:45:06.737322 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":280,"threads_started":1}
I20260620 18:45:06.740656 20749 heartbeater.cc:347] Connected to a master server at 127.5.225.190:36571
I20260620 18:45:06.740988 20749 heartbeater.cc:464] Registering TS with master...
I20260620 18:45:06.741832 20749 heartbeater.cc:511] Master 127.5.225.190:36571 requested a full tablet report, sending...
I20260620 18:45:06.743587 20431 ts_manager.cc:194] Registered new tserver with Master: 1d6dbe1a84d64c37be48f3f9acffbc62 (127.5.225.130:40447)
I20260620 18:45:06.744988 20431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.130@KRBTEST.COM'} at 127.5.225.130:57545
W20260620 18:45:07.055616 20752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:07.055896 20752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:07.056070 20752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:07.063479 20752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:07.063604 20752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:07.063848 20752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.131
I20260620 18:45:07.073865 20752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:33725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.131:38317
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.225.131
--webserver_port=38251
--tserver_master_addrs=127.5.225.190:36571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.131
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:45:07.075280 20752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:07.076643 20752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:07.090526 20760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:07.090608 20758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:07.090590 20757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:07.091491 20752 server_base.cc:1034] running on GCE node
I20260620 18:45:07.092182 20752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:07.093252 20752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:07.094448 20752 hybrid_clock.cc:648] HybridClock initialized: now 1781981107094390 us; error 47 us; skew 500 ppm
I20260620 18:45:07.094770 20752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:45:07 dist-test-slave-k6lg krb5kdc[19847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981107, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:45:07.098289 20752 init.cc:377] Logged in from keytab as kudu/127.5.225.131@KRBTEST.COM (short username kudu)
I20260620 18:45:07.099996 20752 webserver.cc:460] Webserver started at http://127.5.225.131:38251/ using document root <none> and password file <none>
I20260620 18:45:07.100737 20752 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:07.100886 20752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:07.107088 20752 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 18:45:07.110458 20767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:07.111945 20752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:45:07.112191 20752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/wal
uuid: "f0096274c1464896b4dd5597d0d98e53"
format_stamp: "Formatted at 2026-06-20 18:45:04 on dist-test-slave-k6lg"
server_key: "4bfa26af1be9e3bcdf053e01fbdccc11"
server_key_iv: "166483d8b2e52f8546ddbd68d26433ba"
server_key_version: "encryptionkey@0"
I20260620 18:45:07.113286 20752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:07.131227 20752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:07.136152 20752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:07.136443 20752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:07.137501 20752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:45:07.139166 20752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:45:07.139303 20752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:07.139437 20752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:45:07.139528 20752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:07.175894 20752 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.131:38317
I20260620 18:45:07.175941 20879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.131:38317 every 8 connection(s)
I20260620 18:45:07.177379 20752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
I20260620 18:45:07.187240 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 20752
Jun 20 18:45:07 dist-test-slave-k6lg krb5kdc[19847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981107, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:45:07.198697 20619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:07.180201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:48723 (local address 127.5.225.190:36571)
0620 18:45:07.180405 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:07.180410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:07.181700 (+ 1290us) server_negotiation.cc:407] Connection header received
0620 18:45:07.183636 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:07.183640 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:07.183711 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:07.183803 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:07.185322 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:07.186024 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:07.188155 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:07.188810 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:07.188957 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:07.192514 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:07.192520 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:07.192525 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:07.192565 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:07.195550 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:07.196271 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:07.196282 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:07.196284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:07.196401 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:07.196840 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:07.196858 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:07.196861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:07.197135 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:07.197314 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:07.198013 (+ 699us) server_negotiation.cc:299] Negotiation successful
0620 18:45:07.198326 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 18:45:07.199807 20882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:07.180723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.131:48723)
0620 18:45:07.181520 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:07.181565 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:07.183359 (+ 1794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:07.184017 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:07.184025 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:07.184568 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:07.185134 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:07.185151 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:07.186223 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:07.186231 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:07.188001 (+ 1770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:07.188014 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:07.189064 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:07.189071 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:07.189310 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:07.190029 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:07.190051 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:07.192287 (+ 2236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:45:07.195690 (+ 3403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:07.195699 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:07.195717 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:07.196145 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:07.196511 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:07.196516 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:07.196519 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:07.196718 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:07.197274 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:45:07.197283 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:45:07.197646 (+ 363us) client_negotiation.cc:769] Sending connection context
0620 18:45:07.197923 (+ 277us) client_negotiation.cc:240] Negotiation successful
0620 18:45:07.198486 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":325,"threads_started":1}
I20260620 18:45:07.201268 20880 heartbeater.cc:347] Connected to a master server at 127.5.225.190:36571
I20260620 18:45:07.201659 20880 heartbeater.cc:464] Registering TS with master...
I20260620 18:45:07.202642 20880 heartbeater.cc:511] Master 127.5.225.190:36571 requested a full tablet report, sending...
I20260620 18:45:07.203234 20883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:07.188257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57760 (local address 127.5.225.190:36571)
0620 18:45:07.188833 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:07.188838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:07.189183 (+ 345us) server_negotiation.cc:407] Connection header received
0620 18:45:07.189382 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:07.189389 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:07.189451 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:07.189563 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:07.191173 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:07.192091 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:07.193667 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:07.194238 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:07.194371 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:07.196945 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:07.196950 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:07.196953 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:07.196996 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:07.200157 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:07.201003 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:07.201008 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:07.201012 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:07.201138 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:07.201591 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:07.201598 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:07.201603 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:07.201952 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:07.202294 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:07.202600 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 18:45:07.202837 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":281,"threads_started":1}
I20260620 18:45:07.204664 20431 ts_manager.cc:194] Registered new tserver with Master: f0096274c1464896b4dd5597d0d98e53 (127.5.225.131:38317)
I20260620 18:45:07.205794 20431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.131@KRBTEST.COM'} at 127.5.225.131:48723
I20260620 18:45:07.206063 6022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:45:07.213573 20883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:07.207101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57776 (local address 127.5.225.190:36571)
0620 18:45:07.207307 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:07.207315 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:07.207805 (+ 490us) server_negotiation.cc:407] Connection header received
0620 18:45:07.207998 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:07.208002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:07.208132 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:07.208246 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:45:07.208747 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:07.209483 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:07.211369 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:07.211803 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:07.211900 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:07.212471 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:45:07.212962 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:45:07.213103 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:07.213233 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 18:45:07.213295 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:45:07.214597 20431 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:57776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:45:07.222025 20431 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:45:07.249662 20890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:07.233817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.131:38317 (local address 127.0.0.1:57198)
0620 18:45:07.234894 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:07.234939 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:07.235100 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:07.241326 (+ 6226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:07.241330 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:07.241346 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:45:07.241957 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:07.241969 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:07.243960 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:07.243965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:07.246694 (+ 2729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:07.246708 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:07.248592 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:07.248600 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:07.248791 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:07.248852 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 18:45:07.248969 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 18:45:07.249188 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"mutex_wait_us":271,"thread_start_us":719,"threads_started":1}
I20260620 18:45:07.250130 20894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:07.234319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:37807 (local address 127.0.0.1:46306)
0620 18:45:07.239989 (+ 5670us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:07.240019 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:07.240156 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:07.241124 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:07.241132 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:07.241161 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:45:07.241868 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:07.241887 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:07.244042 (+ 2155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:07.244048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:07.246701 (+ 2653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:07.246717 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:07.248956 (+ 2239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:07.248963 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:07.249159 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:07.249192 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 18:45:07.249287 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 18:45:07.249415 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5503,"mutex_wait_us":209,"thread_start_us":5663,"threads_started":1}
I20260620 18:45:07.250753 20892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:07.235316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.130:40447 (local address 127.0.0.1:39940)
0620 18:45:07.240205 (+ 4889us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:07.240223 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:07.240320 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:07.241297 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:07.241304 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:07.241334 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:45:07.242137 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:07.242180 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:07.243753 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:07.243760 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:07.247680 (+ 3920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:07.247695 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:07.249839 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:07.249854 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:07.250022 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:07.250052 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 18:45:07.250239 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 18:45:07.250365 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4757,"mutex_wait_us":198,"thread_start_us":4319,"threads_started":1}
I20260620 18:45:07.251185 20891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:07.234138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57198 (local address 127.5.225.131:38317)
0620 18:45:07.240780 (+ 6642us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:07.240787 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:07.240820 (+ 33us) server_negotiation.cc:407] Connection header received
0620 18:45:07.240931 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:07.240942 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:07.241139 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:07.241426 (+ 287us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:45:07.242186 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:07.243807 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:07.246884 (+ 3077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:07.248364 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:07.250303 (+ 1939us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:07.250584 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:07.250658 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 18:45:07.250840 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6484,"thread_start_us":6328,"threads_started":1}
I20260620 18:45:07.251518 20893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:07.234919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46306 (local address 127.5.225.129:37807)
0620 18:45:07.240550 (+ 5631us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:07.240560 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:07.240589 (+ 29us) server_negotiation.cc:407] Connection header received
0620 18:45:07.240695 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:07.240704 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:07.240889 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:07.241121 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:45:07.242144 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:07.243864 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:07.246895 (+ 3031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:07.248403 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:07.248563 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:07.248798 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:07.250629 (+ 1831us) server_negotiation.cc:299] Negotiation successful
0620 18:45:07.250798 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5483,"thread_start_us":5363,"threads_started":1}
I20260620 18:45:07.252063 20895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:07.235965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39940 (local address 127.5.225.130:40447)
0620 18:45:07.236570 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:07.236582 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:07.240310 (+ 3728us) server_negotiation.cc:407] Connection header received
0620 18:45:07.240532 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:07.240543 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:07.240820 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:07.240993 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:45:07.242394 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:07.243623 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:07.247879 (+ 4256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:07.249281 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:07.249393 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:07.249597 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:07.251010 (+ 1413us) server_negotiation.cc:299] Negotiation successful
0620 18:45:07.251740 (+ 730us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":290,"threads_started":1}
I20260620 18:45:07.254940 20815 tablet_service.cc:1467] Processing CreateTablet for tablet 26a6276abeec4f2dad6728d1c99b1e59 (DEFAULT_TABLE table=test-table [id=68eae014b3194c53a1543f2505940831]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:45:07.255518 20552 tablet_service.cc:1467] Processing CreateTablet for tablet 26a6276abeec4f2dad6728d1c99b1e59 (DEFAULT_TABLE table=test-table [id=68eae014b3194c53a1543f2505940831]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:45:07.256367 20815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26a6276abeec4f2dad6728d1c99b1e59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:45:07.256700 20552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26a6276abeec4f2dad6728d1c99b1e59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:45:07.264403 20684 tablet_service.cc:1467] Processing CreateTablet for tablet 26a6276abeec4f2dad6728d1c99b1e59 (DEFAULT_TABLE table=test-table [id=68eae014b3194c53a1543f2505940831]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:45:07.265865 20684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26a6276abeec4f2dad6728d1c99b1e59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:45:07.269469 20896 tablet_bootstrap.cc:492] T 26a6276abeec4f2dad6728d1c99b1e59 P f0096274c1464896b4dd5597d0d98e53: Bootstrap starting.
I20260620 18:45:07.272732 20897 tablet_bootstrap.cc:492] T 26a6276abeec4f2dad6728d1c99b1e59 P 5d9e32d518e743f5bc47654c76890cf0: Bootstrap starting.
I20260620 18:45:07.274048 20896 tablet_bootstrap.cc:654] T 26a6276abeec4f2dad6728d1c99b1e59 P f0096274c1464896b4dd5597d0d98e53: Neither blocks nor log segments found. Creating new log.
I20260620 18:45:07.276036 20896 log.cc:824] T 26a6276abeec4f2dad6728d1c99b1e59 P f0096274c1464896b4dd5597d0d98e53: Log is configured to *not* fsync() on all Append() calls
I20260620 18:45:07.276104 20897 tablet_bootstrap.cc:654] T 26a6276abeec4f2dad6728d1c99b1e59 P 5d9e32d518e743f5bc47654c76890cf0: Neither blocks nor log segments found. Creating new log.
I20260620 18:45:07.277405 20897 log.cc:824] T 26a6276abeec4f2dad6728d1c99b1e59 P 5d9e32d518e743f5bc47654c76890cf0: Log is configured to *not* fsync() on all Append() calls
I20260620 18:45:07.279592 20896 tablet_bootstrap.cc:492] T 26a6276abeec4f2dad6728d1c99b1e59 P f0096274c1464896b4dd5597d0d98e53: No bootstrap required, opened a new log
I20260620 18:45:07.279899 20896 ts_tablet_manager.cc:1397] T 26a6276abeec4f2dad6728d1c99b1e59 P f0096274c1464896b4dd5597d0d98e53: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260620 18:45:07.280145 20897 tablet_bootstrap.cc:492] T 26a6276abeec4f2dad6728d1c99b1e59 P 5d9e32d518e743f5bc47654c76890cf0: No bootstrap required, opened a new log
I20260620 18:45:07.280408 20897 ts_tablet_manager.cc:1397] T 26a6276abeec4f2dad6728d1c99b1e59 P 5d9e32d518e743f5bc47654c76890cf0: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 18:45:07.281759 20900 tablet_bootstrap.cc:492] T 26a6276abeec4f2dad6728d1c99b1e59 P 1d6dbe1a84d64c37be48f3f9acffbc62: Bootstrap starting.
I20260620 18:45:07.285238 20617 heartbeater.cc:503] Master 127.5.225.190:36571 was elected leader, sending a full tablet report...
I20260620 18:45:07.287142 20900 tablet_bootstrap.cc:654] T 26a6276abeec4f2dad6728d1c99b1e59 P 1d6dbe1a84d64c37be48f3f9acffbc62: Neither blocks nor log segments found. Creating new log.
I20260620 18:45:07.288538 20900 log.cc:824] T 26a6276abeec4f2dad6728d1c99b1e59 P 1d6dbe1a84d64c37be48f3f9acffbc62: Log is configured to *not* fsync() on all Append() calls
I20260620 18:45:07.289796 20896 raft_consensus.cc:348] T 26a6276abeec4f2dad6728d1c99b1e59 P f0096274c1464896b4dd5597d0d98e53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0096274c1464896b4dd5597d0d98e53" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 38317 } } peers { permanent_uuid: "5d9e32d518e743f5bc47654c76890cf0" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37807 } } peers { permanent_uuid: "1d6dbe1a84d64c37be48f3f9acffbc62" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40447 } }
I20260620 18:45:07.290211 20896 raft_consensus.cc:374] T 26a6276abeec4f2dad6728d1c99b1e59 P f0096274c1464896b4dd5597d0d98e53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:45:07.290309 20896 raft_consensus.cc:729] T 26a6276abeec4f2dad6728d1c99b1e59 P f0096274c1464896b4dd5597d0d98e53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0096274c1464896b4dd5597d0d98e53, State: Initialized, Role: FOLLOWER
I20260620 18:45:07.290948 20896 consensus_queue.cc:260] T 26a6276abeec4f2dad6728d1c99b1e59 P f0096274c1464896b4dd5597d0d98e53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0096274c1464896b4dd5597d0d98e53" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 38317 } } peers { permanent_uuid: "5d9e32d518e743f5bc47654c76890cf0" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37807 } } peers { permanent_uuid: "1d6dbe1a84d64c37be48f3f9acffbc62" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40447 } }
I20260620 18:45:07.291771 20897 raft_consensus.cc:348] T 26a6276abeec4f2dad6728d1c99b1e59 P 5d9e32d518e743f5bc47654c76890cf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0096274c1464896b4dd5597d0d98e53" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 38317 } } peers { permanent_uuid: "5d9e32d518e743f5bc47654c76890cf0" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37807 } } peers { permanent_uuid: "1d6dbe1a84d64c37be48f3f9acffbc62" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40447 } }
I20260620 18:45:07.292300 20897 raft_consensus.cc:374] T 26a6276abeec4f2dad6728d1c99b1e59 P 5d9e32d518e743f5bc47654c76890cf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:45:07.292439 20897 raft_consensus.cc:729] T 26a6276abeec4f2dad6728d1c99b1e59 P 5d9e32d518e743f5bc47654c76890cf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d9e32d518e743f5bc47654c76890cf0, State: Initialized, Role: FOLLOWER
I20260620 18:45:07.293149 20897 consensus_queue.cc:260] T 26a6276abeec4f2dad6728d1c99b1e59 P 5d9e32d518e743f5bc47654c76890cf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0096274c1464896b4dd5597d0d98e53" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 38317 } } peers { permanent_uuid: "5d9e32d518e743f5bc47654c76890cf0" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37807 } } peers { permanent_uuid: "1d6dbe1a84d64c37be48f3f9acffbc62" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40447 } }
I20260620 18:45:07.295426 20897 ts_tablet_manager.cc:1428] T 26a6276abeec4f2dad6728d1c99b1e59 P 5d9e32d518e743f5bc47654c76890cf0: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
I20260620 18:45:07.297323 20880 heartbeater.cc:503] Master 127.5.225.190:36571 was elected leader, sending a full tablet report...
I20260620 18:45:07.298280 20896 ts_tablet_manager.cc:1428] T 26a6276abeec4f2dad6728d1c99b1e59 P f0096274c1464896b4dd5597d0d98e53: Time spent starting tablet: real 0.018s user 0.011s sys 0.009s
I20260620 18:45:07.299420 20900 tablet_bootstrap.cc:492] T 26a6276abeec4f2dad6728d1c99b1e59 P 1d6dbe1a84d64c37be48f3f9acffbc62: No bootstrap required, opened a new log
I20260620 18:45:07.299738 20900 ts_tablet_manager.cc:1397] T 26a6276abeec4f2dad6728d1c99b1e59 P 1d6dbe1a84d64c37be48f3f9acffbc62: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.008s
I20260620 18:45:07.307226 20900 raft_consensus.cc:348] T 26a6276abeec4f2dad6728d1c99b1e59 P 1d6dbe1a84d64c37be48f3f9acffbc62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0096274c1464896b4dd5597d0d98e53" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 38317 } } peers { permanent_uuid: "5d9e32d518e743f5bc47654c76890cf0" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37807 } } peers { permanent_uuid: "1d6dbe1a84d64c37be48f3f9acffbc62" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40447 } }
I20260620 18:45:07.307653 20900 raft_consensus.cc:374] T 26a6276abeec4f2dad6728d1c99b1e59 P 1d6dbe1a84d64c37be48f3f9acffbc62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:45:07.307767 20900 raft_consensus.cc:729] T 26a6276abeec4f2dad6728d1c99b1e59 P 1d6dbe1a84d64c37be48f3f9acffbc62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d6dbe1a84d64c37be48f3f9acffbc62, State: Initialized, Role: FOLLOWER
I20260620 18:45:07.308349 20900 consensus_queue.cc:260] T 26a6276abeec4f2dad6728d1c99b1e59 P 1d6dbe1a84d64c37be48f3f9acffbc62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0096274c1464896b4dd5597d0d98e53" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 38317 } } peers { permanent_uuid: "5d9e32d518e743f5bc47654c76890cf0" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37807 } } peers { permanent_uuid: "1d6dbe1a84d64c37be48f3f9acffbc62" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40447 } }
I20260620 18:45:07.309530 20749 heartbeater.cc:503] Master 127.5.225.190:36571 was elected leader, sending a full tablet report...
I20260620 18:45:07.310086 20900 ts_tablet_manager.cc:1428] T 26a6276abeec4f2dad6728d1c99b1e59 P 1d6dbe1a84d64c37be48f3f9acffbc62: Time spent starting tablet: real 0.010s user 0.004s sys 0.006s
W20260620 18:45:07.429174 20881 tablet.cc:2369] T 26a6276abeec4f2dad6728d1c99b1e59 P f0096274c1464896b4dd5597d0d98e53: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:45:07.465396 20750 tablet.cc:2369] T 26a6276abeec4f2dad6728d1c99b1e59 P 1d6dbe1a84d64c37be48f3f9acffbc62: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:45:07.500356 20618 tablet.cc:2369] T 26a6276abeec4f2dad6728d1c99b1e59 P 5d9e32d518e743f5bc47654c76890cf0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:45:07.574100 20902 raft_consensus.cc:482] T 26a6276abeec4f2dad6728d1c99b1e59 P f0096274c1464896b4dd5597d0d98e53 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:45:07.574525 20902 raft_consensus.cc:504] T 26a6276abeec4f2dad6728d1c99b1e59 P f0096274c1464896b4dd5597d0d98e53 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0096274c1464896b4dd5597d0d98e53" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 38317 } } peers { permanent_uuid: "5d9e32d518e743f5bc47654c76890cf0" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37807 } } peers { permanent_uuid: "1d6dbe1a84d64c37be48f3f9acffbc62" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40447 } }
I20260620 18:45:07.576333 20902 leader_election.cc:288] T 26a6276abeec4f2dad6728d1c99b1e59 P f0096274c1464896b4dd5597d0d98e53 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d9e32d518e743f5bc47654c76890cf0 (127.5.225.129:37807), 1d6dbe1a84d64c37be48f3f9acffbc62 (127.5.225.130:40447)
I20260620 18:45:07.583655 20895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:07.576858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:45487 (local address 127.5.225.130:40447)
0620 18:45:07.577098 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:07.577108 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:07.577135 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:45:07.577236 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:07.577245 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:07.577348 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:07.577457 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:45:07.577926 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:07.578874 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:07.581454 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:07.582718 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:07.582876 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:07.582956 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:07.583299 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 18:45:07.583420 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 18:45:07.583711 20882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:07.576604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.130:40447 (local address 127.5.225.131:45487)
0620 18:45:07.576849 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:07.576871 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:07.577010 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:07.577513 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:07.577518 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:07.577547 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:45:07.577773 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:07.577786 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:07.579185 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:07.579195 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:07.581290 (+ 2095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:07.581307 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:07.582937 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:07.582942 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:07.583077 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:07.583107 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 18:45:07.583257 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 18:45:07.583402 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":34}
I20260620 18:45:07.584026 20893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:07.577025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:44593 (local address 127.5.225.129:37807)
0620 18:45:07.577246 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:07.577254 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:07.577739 (+ 485us) server_negotiation.cc:407] Connection header received
0620 18:45:07.577925 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:07.577930 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:07.578038 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:07.578135 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:45:07.578753 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:07.579538 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:07.581810 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:07.583081 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:07.583204 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:07.583302 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:07.583634 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 18:45:07.583775 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 18:45:07.584312 20905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:07.576796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:37807 (local address 127.5.225.131:44593)
0620 18:45:07.577621 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:07.577637 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:07.577755 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:07.578251 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:07.578254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:07.578279 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:45:07.578567 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:07.578575 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:07.579709 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:07.579717 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:07.581665 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:07.581675 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:07.583290 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:07.583295 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:07.583459 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:07.583490 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 18:45:07.583585 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 18:45:07.583689 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"mutex_wait_us":1,"thread_start_us":475,"threads_started":1}
I20260620 18:45:07.584904 20704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a6276abeec4f2dad6728d1c99b1e59" candidate_uuid: "f0096274c1464896b4dd5597d0d98e53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d6dbe1a84d64c37be48f3f9acffbc62" is_pre_election: true
I20260620 18:45:07.585426 20704 raft_consensus.cc:2454] T 26a6276abeec4f2dad6728d1c99b1e59 P 1d6dbe1a84d64c37be48f3f9acffbc62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0096274c1464896b4dd5597d0d98e53 in term 0.
I20260620 18:45:07.585420 20572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a6276abeec4f2dad6728d1c99b1e59" candidate_uuid: "f0096274c1464896b4dd5597d0d98e53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d9e32d518e743f5bc47654c76890cf0" is_pre_election: true
I20260620 18:45:07.586025 20572 raft_consensus.cc:2454] T 26a6276abeec4f2dad6728d1c99b1e59 P 5d9e32d518e743f5bc47654c76890cf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0096274c1464896b4dd5597d0d98e53 in term 0.
I20260620 18:45:07.586207 20770 leader_election.cc:302] T 26a6276abeec4f2dad6728d1c99b1e59 P f0096274c1464896b4dd5597d0d98e53 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d6dbe1a84d64c37be48f3f9acffbc62, f0096274c1464896b4dd5597d0d98e53; no voters:
I20260620 18:45:07.586658 20902 raft_consensus.cc:2781] T 26a6276abeec4f2dad6728d1c99b1e59 P f0096274c1464896b4dd5597d0d98e53 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:45:07.586813 20902 raft_consensus.cc:482] T 26a6276abeec4f2dad6728d1c99b1e59 P f0096274c1464896b4dd5597d0d98e53 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:45:07.586928 20902 raft_consensus.cc:3037] T 26a6276abeec4f2dad6728d1c99b1e59 P f0096274c1464896b4dd5597d0d98e53 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:45:07.589135 20902 raft_consensus.cc:504] T 26a6276abeec4f2dad6728d1c99b1e59 P f0096274c1464896b4dd5597d0d98e53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0096274c1464896b4dd5597d0d98e53" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 38317 } } peers { permanent_uuid: "5d9e32d518e743f5bc47654c76890cf0" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37807 } } peers { permanent_uuid: "1d6dbe1a84d64c37be48f3f9acffbc62" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40447 } }
I20260620 18:45:07.589800 20902 leader_election.cc:288] T 26a6276abeec4f2dad6728d1c99b1e59 P f0096274c1464896b4dd5597d0d98e53 [CANDIDATE]: Term 1 election: Requested vote from peers 5d9e32d518e743f5bc47654c76890cf0 (127.5.225.129:37807), 1d6dbe1a84d64c37be48f3f9acffbc62 (127.5.225.130:40447)
I20260620 18:45:07.590242 20572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a6276abeec4f2dad6728d1c99b1e59" candidate_uuid: "f0096274c1464896b4dd5597d0d98e53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d9e32d518e743f5bc47654c76890cf0"
I20260620 18:45:07.590389 20704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a6276abeec4f2dad6728d1c99b1e59" candidate_uuid: "f0096274c1464896b4dd5597d0d98e53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d6dbe1a84d64c37be48f3f9acffbc62"
I20260620 18:45:07.590466 20572 raft_consensus.cc:3037] T 26a6276abeec4f2dad6728d1c99b1e59 P 5d9e32d518e743f5bc47654c76890cf0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:45:07.590636 20704 raft_consensus.cc:3037] T 26a6276abeec4f2dad6728d1c99b1e59 P 1d6dbe1a84d64c37be48f3f9acffbc62 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:45:07.593053 20704 raft_consensus.cc:2454] T 26a6276abeec4f2dad6728d1c99b1e59 P 1d6dbe1a84d64c37be48f3f9acffbc62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0096274c1464896b4dd5597d0d98e53 in term 1.
I20260620 18:45:07.593178 20572 raft_consensus.cc:2454] T 26a6276abeec4f2dad6728d1c99b1e59 P 5d9e32d518e743f5bc47654c76890cf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0096274c1464896b4dd5597d0d98e53 in term 1.
I20260620 18:45:07.593475 20770 leader_election.cc:302] T 26a6276abeec4f2dad6728d1c99b1e59 P f0096274c1464896b4dd5597d0d98e53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d6dbe1a84d64c37be48f3f9acffbc62, f0096274c1464896b4dd5597d0d98e53; no voters:
I20260620 18:45:07.593739 20902 raft_consensus.cc:2781] T 26a6276abeec4f2dad6728d1c99b1e59 P f0096274c1464896b4dd5597d0d98e53 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:45:07.594347 20902 raft_consensus.cc:686] T 26a6276abeec4f2dad6728d1c99b1e59 P f0096274c1464896b4dd5597d0d98e53 [term 1 LEADER]: Becoming Leader. State: Replica: f0096274c1464896b4dd5597d0d98e53, State: Running, Role: LEADER
I20260620 18:45:07.594907 20902 consensus_queue.cc:237] T 26a6276abeec4f2dad6728d1c99b1e59 P f0096274c1464896b4dd5597d0d98e53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0096274c1464896b4dd5597d0d98e53" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 38317 } } peers { permanent_uuid: "5d9e32d518e743f5bc47654c76890cf0" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37807 } } peers { permanent_uuid: "1d6dbe1a84d64c37be48f3f9acffbc62" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40447 } }
I20260620 18:45:07.599536 20431 catalog_manager.cc:5495] T 26a6276abeec4f2dad6728d1c99b1e59 P f0096274c1464896b4dd5597d0d98e53 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0096274c1464896b4dd5597d0d98e53 (127.5.225.131). New cstate: current_term: 1 leader_uuid: "f0096274c1464896b4dd5597d0d98e53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0096274c1464896b4dd5597d0d98e53" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 38317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d9e32d518e743f5bc47654c76890cf0" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d6dbe1a84d64c37be48f3f9acffbc62" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40447 } health_report { overall_health: UNKNOWN } } }
I20260620 18:45:07.703598 20891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:07.697838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57214 (local address 127.5.225.131:38317)
0620 18:45:07.698043 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:07.698050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:07.698070 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:45:07.698259 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:07.698263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:07.698351 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:07.698512 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:45:07.698877 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:07.699734 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:07.701154 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:07.701571 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:07.701716 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:07.702330 (+ 614us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:45:07.702837 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:45:07.703004 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:07.703145 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 18:45:07.703247 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:45:07.711900 20704 raft_consensus.cc:1261] T 26a6276abeec4f2dad6728d1c99b1e59 P 1d6dbe1a84d64c37be48f3f9acffbc62 [term 1 FOLLOWER]: Refusing update from remote peer f0096274c1464896b4dd5597d0d98e53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:45:07.712108 20572 raft_consensus.cc:1261] T 26a6276abeec4f2dad6728d1c99b1e59 P 5d9e32d518e743f5bc47654c76890cf0 [term 1 FOLLOWER]: Refusing update from remote peer f0096274c1464896b4dd5597d0d98e53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:45:07.713006 20906 consensus_queue.cc:1035] T 26a6276abeec4f2dad6728d1c99b1e59 P f0096274c1464896b4dd5597d0d98e53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d6dbe1a84d64c37be48f3f9acffbc62" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:45:07.713388 20902 consensus_queue.cc:1035] T 26a6276abeec4f2dad6728d1c99b1e59 P f0096274c1464896b4dd5597d0d98e53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d9e32d518e743f5bc47654c76890cf0" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 37807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:45:07.744660 20911 mvcc.cc:205] Tried to move back new op lower bound from 7298994617177350144 to 7298994616712421376. Current Snapshot: MvccSnapshot[applied={T|T < 7298994617177350144}]
I20260620 18:45:07.763494 20431 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:57776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:45:07.763813 20431 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:57776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:45:07.767586 20431 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1: Sending DeleteTablet for 3 replicas of tablet 26a6276abeec4f2dad6728d1c99b1e59
I20260620 18:45:07.768864 20815 tablet_service.cc:1514] Processing DeleteTablet for tablet 26a6276abeec4f2dad6728d1c99b1e59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:45:07 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:57198
I20260620 18:45:07.768851 20552 tablet_service.cc:1514] Processing DeleteTablet for tablet 26a6276abeec4f2dad6728d1c99b1e59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:45:07 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:46306
I20260620 18:45:07.769209 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 20468
I20260620 18:45:07.769881 20684 tablet_service.cc:1514] Processing DeleteTablet for tablet 26a6276abeec4f2dad6728d1c99b1e59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:45:07 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:39940
I20260620 18:45:07.778746 20922 tablet_replica.cc:330] T 26a6276abeec4f2dad6728d1c99b1e59 P 1d6dbe1a84d64c37be48f3f9acffbc62: stopping tablet replica
I20260620 18:45:07.779362 20922 raft_consensus.cc:2229] T 26a6276abeec4f2dad6728d1c99b1e59 P 1d6dbe1a84d64c37be48f3f9acffbc62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:45:07.780030 20922 raft_consensus.cc:2258] T 26a6276abeec4f2dad6728d1c99b1e59 P 1d6dbe1a84d64c37be48f3f9acffbc62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:45:07.783130 20922 ts_tablet_manager.cc:1905] T 26a6276abeec4f2dad6728d1c99b1e59 P 1d6dbe1a84d64c37be48f3f9acffbc62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:45:07.786715 20919 tablet_replica.cc:330] T 26a6276abeec4f2dad6728d1c99b1e59 P f0096274c1464896b4dd5597d0d98e53: stopping tablet replica
I20260620 18:45:07.787191 20919 raft_consensus.cc:2229] T 26a6276abeec4f2dad6728d1c99b1e59 P f0096274c1464896b4dd5597d0d98e53 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:45:07.787945 20919 raft_consensus.cc:2258] T 26a6276abeec4f2dad6728d1c99b1e59 P f0096274c1464896b4dd5597d0d98e53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:45:07.790035 20920 tablet_replica.cc:330] T 26a6276abeec4f2dad6728d1c99b1e59 P 5d9e32d518e743f5bc47654c76890cf0: stopping tablet replica
I20260620 18:45:07.790452 20920 raft_consensus.cc:2229] T 26a6276abeec4f2dad6728d1c99b1e59 P 5d9e32d518e743f5bc47654c76890cf0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:45:07.790957 20920 raft_consensus.cc:2258] T 26a6276abeec4f2dad6728d1c99b1e59 P 5d9e32d518e743f5bc47654c76890cf0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:45:07.791970 20922 ts_tablet_manager.cc:1918] T 26a6276abeec4f2dad6728d1c99b1e59 P 1d6dbe1a84d64c37be48f3f9acffbc62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:45:07.792186 20922 log.cc:1196] T 26a6276abeec4f2dad6728d1c99b1e59 P 1d6dbe1a84d64c37be48f3f9acffbc62: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/wal/wals/26a6276abeec4f2dad6728d1c99b1e59
I20260620 18:45:07.792737 20922 ts_tablet_manager.cc:1939] T 26a6276abeec4f2dad6728d1c99b1e59 P 1d6dbe1a84d64c37be48f3f9acffbc62: Deleting consensus metadata
I20260620 18:45:07.792860 20919 ts_tablet_manager.cc:1905] T 26a6276abeec4f2dad6728d1c99b1e59 P f0096274c1464896b4dd5597d0d98e53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:45:07.794481 20920 ts_tablet_manager.cc:1905] T 26a6276abeec4f2dad6728d1c99b1e59 P 5d9e32d518e743f5bc47654c76890cf0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:45:07.794657 20417 catalog_manager.cc:4841] TS 1d6dbe1a84d64c37be48f3f9acffbc62 (127.5.225.130:40447): tablet 26a6276abeec4f2dad6728d1c99b1e59 (table test-table [id=68eae014b3194c53a1543f2505940831]) successfully deleted
I20260620 18:45:07.800228 20919 ts_tablet_manager.cc:1918] T 26a6276abeec4f2dad6728d1c99b1e59 P f0096274c1464896b4dd5597d0d98e53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:45:07.800412 20919 log.cc:1196] T 26a6276abeec4f2dad6728d1c99b1e59 P f0096274c1464896b4dd5597d0d98e53: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/wal/wals/26a6276abeec4f2dad6728d1c99b1e59
I20260620 18:45:07.800925 20919 ts_tablet_manager.cc:1939] T 26a6276abeec4f2dad6728d1c99b1e59 P f0096274c1464896b4dd5597d0d98e53: Deleting consensus metadata
I20260620 18:45:07.802821 20417 catalog_manager.cc:4841] TS f0096274c1464896b4dd5597d0d98e53 (127.5.225.131:38317): tablet 26a6276abeec4f2dad6728d1c99b1e59 (table test-table [id=68eae014b3194c53a1543f2505940831]) successfully deleted
I20260620 18:45:07.803392 20920 ts_tablet_manager.cc:1918] T 26a6276abeec4f2dad6728d1c99b1e59 P 5d9e32d518e743f5bc47654c76890cf0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:45:07.803629 20920 log.cc:1196] T 26a6276abeec4f2dad6728d1c99b1e59 P 5d9e32d518e743f5bc47654c76890cf0: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/wal/wals/26a6276abeec4f2dad6728d1c99b1e59
I20260620 18:45:07.804188 20920 ts_tablet_manager.cc:1939] T 26a6276abeec4f2dad6728d1c99b1e59 P 5d9e32d518e743f5bc47654c76890cf0: Deleting consensus metadata
I20260620 18:45:07.806031 20418 catalog_manager.cc:4841] TS 5d9e32d518e743f5bc47654c76890cf0 (127.5.225.129:37807): tablet 26a6276abeec4f2dad6728d1c99b1e59 (table test-table [id=68eae014b3194c53a1543f2505940831]) successfully deleted
I20260620 18:45:07.808177 20893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:07.770235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46310 (local address 127.5.225.129:37807)
0620 18:45:07.770413 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:07.770418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:07.788896 (+ 18478us) server_negotiation.cc:407] Connection header received
0620 18:45:07.789110 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:07.789115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:07.789220 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:07.789332 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:07.791933 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:07.793021 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:07.795273 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:07.795890 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:07.796034 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:07.800012 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:07.800032 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:07.800053 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:07.800120 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:07.804793 (+ 4673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:07.805730 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:07.805740 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:07.805744 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:07.805932 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:07.806445 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:07.806454 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:07.806459 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:07.806958 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:07.807285 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:07.807539 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 18:45:07.807778 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:45:07.808902 20612 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:07.904141 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 20468
I20260620 18:45:07.923273 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 20621
I20260620 18:45:07.939318 20895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:07.924381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39942 (local address 127.5.225.130:40447)
0620 18:45:07.924603 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:07.924610 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:07.924632 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:45:07.924702 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:07.924709 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:07.924798 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:07.924912 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:07.926222 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:07.927376 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:07.928787 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:07.929429 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:07.929561 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:07.932279 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:07.932306 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:07.932326 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:07.932368 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:07.936036 (+ 3668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:07.936777 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:07.936788 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:07.936791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:07.936960 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:07.937330 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:07.937335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:07.937338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:07.937802 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:07.938143 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:07.938684 (+ 541us) server_negotiation.cc:299] Negotiation successful
0620 18:45:07.938911 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 18:45:07.940073 20744 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:08.034898 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 20621
I20260620 18:45:08.058617 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 20752
I20260620 18:45:08.077891 20891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:08.059474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57218 (local address 127.5.225.131:38317)
0620 18:45:08.059684 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:08.059691 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:08.059710 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:45:08.059898 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:08.059904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:08.060026 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:08.060142 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:08.061564 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:08.062503 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:08.063886 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:08.064348 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:08.064506 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:08.071443 (+ 6937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:08.071457 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:08.071473 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:08.071518 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:08.074950 (+ 3432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:08.075716 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:08.075727 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:08.075729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:08.075885 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:08.076276 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:08.076282 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:08.076285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:08.076640 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:08.076980 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:08.077246 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 18:45:08.077478 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:45:08.078620 20875 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:08.200583 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 20752
I20260620 18:45:08.220818 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 20398
I20260620 18:45:08.222020 20461 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:08.337342 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 20398
I20260620 18:45:08.358893 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:36571
--webserver_interface=127.5.225.190
--webserver_port=44363
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:33725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:36571
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:45:08.691089 20927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:08.691440 20927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:08.691601 20927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:08.698848 20927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:45:08.698982 20927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:08.699092 20927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:08.699189 20927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:45:08.699275 20927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:45:08.709179 20927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:33725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:36571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:36571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=44363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:45:08.710690 20927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:08.711974 20927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:08.724148 20932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:08.724242 20933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:08.724242 20935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:08.725489 20927 server_base.cc:1034] running on GCE node
I20260620 18:45:08.726408 20927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:08.727955 20927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:08.729187 20927 hybrid_clock.cc:648] HybridClock initialized: now 1781981108729131 us; error 35 us; skew 500 ppm
I20260620 18:45:08.729578 20927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:45:08 dist-test-slave-k6lg krb5kdc[19847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981108, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:45:08.733297 20927 init.cc:377] Logged in from keytab as kudu/127.5.225.190@KRBTEST.COM (short username kudu)
I20260620 18:45:08.735035 20927 webserver.cc:460] Webserver started at http://127.5.225.190:44363/ using document root <none> and password file <none>
I20260620 18:45:08.735790 20927 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:08.735954 20927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:08.741673 20927 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 18:45:08.744737 20942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:08.746097 20927 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:45:08.746318 20927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "0db553c30c5c482090e33c6691ea0cf1"
format_stamp: "Formatted at 2026-06-20 18:45:02 on dist-test-slave-k6lg"
server_key: "377e7c46fcd3879fc81706dbd2a3a120"
server_key_iv: "a611f987e6e52c3baa4acc419608205b"
server_key_version: "encryptionkey@0"
I20260620 18:45:08.747416 20927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:08.777454 20927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:08.782218 20927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:08.782537 20927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:08.802486 20927 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:36571
I20260620 18:45:08.802508 20993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:36571 every 8 connection(s)
I20260620 18:45:08.804123 20927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:45:08.811203 20994 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:45:08.813299 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 20927
I20260620 18:45:08.814329 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.129:37807
--local_ip_for_outbound_sockets=127.5.225.129
--tserver_master_addrs=127.5.225.190:36571
--webserver_port=34271
--webserver_interface=127.5.225.129
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:33725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:45:08.815667 20994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1: Bootstrap starting.
I20260620 18:45:08.838585 20994 log.cc:824] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1: Log is configured to *not* fsync() on all Append() calls
I20260620 18:45:08.855786 20994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:45:08.856381 20994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1: Bootstrap complete.
I20260620 18:45:08.867420 20994 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0db553c30c5c482090e33c6691ea0cf1" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36571 } }
I20260620 18:45:08.868366 20994 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0db553c30c5c482090e33c6691ea0cf1, State: Initialized, Role: FOLLOWER
I20260620 18:45:08.868986 20994 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0db553c30c5c482090e33c6691ea0cf1" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36571 } }
I20260620 18:45:08.869236 20994 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:45:08.869369 20994 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:45:08.869568 20994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [term 2 FOLLOWER]: Advancing to term 3
I20260620 18:45:08.873649 20994 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0db553c30c5c482090e33c6691ea0cf1" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36571 } }
I20260620 18:45:08.874218 20994 leader_election.cc:302] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0db553c30c5c482090e33c6691ea0cf1; no voters:
I20260620 18:45:08.874925 20994 leader_election.cc:288] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 18:45:08.875082 20998 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 18:45:08.876324 20998 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [term 3 LEADER]: Becoming Leader. State: Replica: 0db553c30c5c482090e33c6691ea0cf1, State: Running, Role: LEADER
I20260620 18:45:08.876886 20998 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0db553c30c5c482090e33c6691ea0cf1" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36571 } }
I20260620 18:45:08.877287 20994 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:45:08.879606 21000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0db553c30c5c482090e33c6691ea0cf1. Latest consensus state: current_term: 3 leader_uuid: "0db553c30c5c482090e33c6691ea0cf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0db553c30c5c482090e33c6691ea0cf1" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36571 } } }
I20260620 18:45:08.879714 20999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0db553c30c5c482090e33c6691ea0cf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0db553c30c5c482090e33c6691ea0cf1" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36571 } } }
I20260620 18:45:08.880036 21000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [sys.catalog]: This master's current role is: LEADER
I20260620 18:45:08.880045 20999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0db553c30c5c482090e33c6691ea0cf1 [sys.catalog]: This master's current role is: LEADER
I20260620 18:45:08.880960 21004 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:45:08.897024 21004 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:45:08.898620 21004 catalog_manager.cc:1259] Loaded cluster ID: ed66c37ebf06469f8c2e7e052ccb0be9
I20260620 18:45:08.898830 21004 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 18:45:08.900971 21004 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc18e58a9ad google::LogMessage::Fail() at ??:0
@ 0x7fc18e58d88a google::LogMessage::SendToLog() at ??:0
@ 0x7fc18e58a4de google::LogMessage::Flush() at ??:0
@ 0x7fc18e58ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc1a3fbd512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fc1a3fba976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
I20260620 18:45:08.934700 21017 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7fc1a4042610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fc1a4042439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fc1911d6345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc1911d991c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fc1911d9799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fc1911b4eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc19448e6db start_thread at ??:0
@ 0x7fc18d42461f clone at ??:0
W20260620 18:45:09.208227 20996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:09.208590 20996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:09.208776 20996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:09.216141 20996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:09.216288 20996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:09.216481 20996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.129
I20260620 18:45:09.226562 20996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:33725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.129:37807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.225.129
--webserver_port=34271
--tserver_master_addrs=127.5.225.190:36571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.129
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:45:09.227911 20996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:09.229261 20996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:09.243067 21035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:09.243117 21036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:09.243145 21038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:09.243958 20996 server_base.cc:1034] running on GCE node
I20260620 18:45:09.244551 20996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:09.245469 20996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:09.246668 20996 hybrid_clock.cc:648] HybridClock initialized: now 1781981109246605 us; error 50 us; skew 500 ppm
I20260620 18:45:09.246982 20996 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:45:09 dist-test-slave-k6lg krb5kdc[19847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981109, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:45:09.254024 20996 init.cc:377] Logged in from keytab as kudu/127.5.225.129@KRBTEST.COM (short username kudu)
I20260620 18:45:09.255551 20996 webserver.cc:460] Webserver started at http://127.5.225.129:34271/ using document root <none> and password file <none>
I20260620 18:45:09.256228 20996 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:09.256330 20996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:09.261725 20996 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20260620 18:45:09.264575 21045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:09.265982 20996 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 18:45:09.266256 20996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/wal
uuid: "5d9e32d518e743f5bc47654c76890cf0"
format_stamp: "Formatted at 2026-06-20 18:45:03 on dist-test-slave-k6lg"
server_key: "90a1d2725838bc673568bdb57dcc13fb"
server_key_iv: "26017eb591c73e7cbdea85d08de7f482"
server_key_version: "encryptionkey@0"
I20260620 18:45:09.267277 20996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:09.298580 20996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:09.303341 20996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:09.303658 20996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:09.304713 20996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:45:09.306126 20996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:45:09.306280 20996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 18:45:09.306430 20996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:45:09.306547 20996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:09.342383 20996 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.129:37807
I20260620 18:45:09.342469 21157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.129:37807 every 8 connection(s)
I20260620 18:45:09.343875 20996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
I20260620 18:45:09.348870 21160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:09.346976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.129:36779)
0620 18:45:09.347754 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:09.348013 (+ 259us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":315,"threads_started":1}
W20260620 18:45:09.349637 21158 heartbeater.cc:644] Failed to heartbeat to 127.5.225.190:36571 (0 consecutive failures): Network error: Failed to ping master at 127.5.225.190:36571: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
I20260620 18:45:09.350535 21160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:09.350154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.129:42499)
0620 18:45:09.350370 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:09.350435 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 18:45:09.351425 21160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:09.351077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.129:53943)
0620 18:45:09.351221 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:09.351301 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 18:45:09.351670 21158 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 18:45:09.352567 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 20996
I20260620 18:45:09.353255 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.130:40447
--local_ip_for_outbound_sockets=127.5.225.130
--tserver_master_addrs=127.5.225.190:36571
--webserver_port=33491
--webserver_interface=127.5.225.130
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:33725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:45:09.684295 21161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:09.684597 21161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:09.684770 21161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:09.693173 21161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:09.693285 21161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:09.693471 21161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.130
I20260620 18:45:09.703452 21161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:33725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.130:40447
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.225.130
--webserver_port=33491
--tserver_master_addrs=127.5.225.190:36571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.130
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:45:09.704814 21161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:09.706216 21161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:09.719893 21167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:09.720139 21169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:09.720219 21166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:09.721189 21161 server_base.cc:1034] running on GCE node
I20260620 18:45:09.721767 21161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:09.722777 21161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:09.724064 21161 hybrid_clock.cc:648] HybridClock initialized: now 1781981109723949 us; error 41 us; skew 500 ppm
I20260620 18:45:09.724376 21161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:45:09 dist-test-slave-k6lg krb5kdc[19847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981109, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:45:09.727682 21161 init.cc:377] Logged in from keytab as kudu/127.5.225.130@KRBTEST.COM (short username kudu)
I20260620 18:45:09.729153 21161 webserver.cc:460] Webserver started at http://127.5.225.130:33491/ using document root <none> and password file <none>
I20260620 18:45:09.729878 21161 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:09.730012 21161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:09.735487 21161 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 18:45:09.738272 21176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:09.739670 21161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:45:09.739861 21161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/wal
uuid: "1d6dbe1a84d64c37be48f3f9acffbc62"
format_stamp: "Formatted at 2026-06-20 18:45:03 on dist-test-slave-k6lg"
server_key: "648ff48894e97290104f95b1831a6c82"
server_key_iv: "958520541ef8ad8fc68074b161ca3f18"
server_key_version: "encryptionkey@0"
I20260620 18:45:09.740854 21161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:09.762816 21161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:09.767042 21161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:09.767333 21161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:09.768366 21161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:45:09.769786 21161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:45:09.769913 21161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:09.770062 21161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:45:09.770164 21161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:09.805092 21161 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.130:40447
I20260620 18:45:09.805146 21288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.130:40447 every 8 connection(s)
I20260620 18:45:09.806591 21161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
I20260620 18:45:09.811702 21291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:09.809726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.130:53389)
0620 18:45:09.810624 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:09.810882 (+ 258us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":395,"threads_started":1}
W20260620 18:45:09.812474 21289 heartbeater.cc:644] Failed to heartbeat to 127.5.225.190:36571 (0 consecutive failures): Network error: Failed to ping master at 127.5.225.190:36571: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
I20260620 18:45:09.813342 21291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:09.813013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.130:55879)
0620 18:45:09.813158 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:09.813225 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 18:45:09.814360 21291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:09.813940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.130:56435)
0620 18:45:09.814077 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:09.814153 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 18:45:09.814625 21289 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 18:45:09.815928 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 21161
I20260620 18:45:09.816566 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.131:38317
--local_ip_for_outbound_sockets=127.5.225.131
--tserver_master_addrs=127.5.225.190:36571
--webserver_port=38251
--webserver_interface=127.5.225.131
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:33725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:45:10.146111 21292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:10.146418 21292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:10.146579 21292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:10.153868 21292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:10.153976 21292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:10.154177 21292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.131
I20260620 18:45:10.163851 21292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:33725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.131:38317
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.225.131
--webserver_port=38251
--tserver_master_addrs=127.5.225.190:36571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.131
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:45:10.165196 21292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:10.166539 21292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:10.179772 21298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:10.179888 21300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:10.179858 21297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:10.180644 21292 server_base.cc:1034] running on GCE node
I20260620 18:45:10.181221 21292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:10.182126 21292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:10.183439 21292 hybrid_clock.cc:648] HybridClock initialized: now 1781981110183391 us; error 37 us; skew 500 ppm
I20260620 18:45:10.183781 21292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:45:10 dist-test-slave-k6lg krb5kdc[19847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981110, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:45:10.187062 21292 init.cc:377] Logged in from keytab as kudu/127.5.225.131@KRBTEST.COM (short username kudu)
I20260620 18:45:10.188561 21292 webserver.cc:460] Webserver started at http://127.5.225.131:38251/ using document root <none> and password file <none>
I20260620 18:45:10.189231 21292 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:10.189332 21292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:10.194725 21292 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:45:10.197340 21307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:10.198863 21292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:45:10.198997 21292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/wal
uuid: "f0096274c1464896b4dd5597d0d98e53"
format_stamp: "Formatted at 2026-06-20 18:45:04 on dist-test-slave-k6lg"
server_key: "4bfa26af1be9e3bcdf053e01fbdccc11"
server_key_iv: "166483d8b2e52f8546ddbd68d26433ba"
server_key_version: "encryptionkey@0"
I20260620 18:45:10.199920 21292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:10.215709 21292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:10.220252 21292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:10.220482 21292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:10.221426 21292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:45:10.222764 21292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:45:10.222851 21292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:10.222924 21292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:45:10.222982 21292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:10.257714 21292 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.131:38317
I20260620 18:45:10.257834 21419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.131:38317 every 8 connection(s)
I20260620 18:45:10.259442 21292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
I20260620 18:45:10.264406 21422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:10.262592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.131:50491)
0620 18:45:10.263313 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:10.263557 (+ 244us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":268,"threads_started":1}
W20260620 18:45:10.265234 21420 heartbeater.cc:644] Failed to heartbeat to 127.5.225.190:36571 (0 consecutive failures): Network error: Failed to ping master at 127.5.225.190:36571: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
I20260620 18:45:10.266243 21422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:10.265774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.131:48561)
0620 18:45:10.265959 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:10.266060 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 18:45:10.267319 21422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:10.266917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.131:39679)
0620 18:45:10.267128 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:10.267213 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
W20260620 18:45:10.267560 21420 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 18:45:10.269484 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 21292
I20260620 18:45:10.352945 21424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:10.352184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.129:37049)
0620 18:45:10.352610 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:10.352698 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":210,"threads_started":1}
I20260620 18:45:10.815996 21425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:10.815202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.130:40963)
0620 18:45:10.815664 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:10.815755 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":227,"threads_started":1}
I20260620 18:45:11.268958 21426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:11.268105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.131:36251)
0620 18:45:11.268591 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:11.268694 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":228,"threads_started":1}
I20260620 18:45:11.354734 21427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:11.353932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.129:52873)
0620 18:45:11.354497 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:11.354574 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":341,"threads_started":1}
I20260620 18:45:11.817615 21428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:11.816925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.130:53557)
0620 18:45:11.817366 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:11.817445 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":208,"threads_started":1}
I20260620 18:45:12.270852 21429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:12.269952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.131:41037)
0620 18:45:12.270553 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:12.270640 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":322,"threads_started":1}
I20260620 18:45:12.356329 21430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:12.355612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.129:52485)
0620 18:45:12.356063 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:12.356143 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":244,"threads_started":1}
I20260620 18:45:12.819412 21431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:12.818648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.130:33251)
0620 18:45:12.819142 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:12.819230 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":267,"threads_started":1}
I20260620 18:45:13.272394 21432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:13.271694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.131:39209)
0620 18:45:13.272142 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:13.272222 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":238,"threads_started":1}
I20260620 18:45:13.358040 21433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:13.357296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.129:56523)
0620 18:45:13.357743 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:13.357869 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":232,"threads_started":1}
I20260620 18:45:13.821106 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:13.820345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.130:49725)
0620 18:45:13.820822 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:13.820907 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":226,"threads_started":1}
I20260620 18:45:14.274029 21435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:14.273313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.131:36879)
0620 18:45:14.273756 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:14.273835 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":216,"threads_started":1}
I20260620 18:45:14.359696 21436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:14.358975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.129:37585)
0620 18:45:14.359441 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:14.359526 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":250,"threads_started":1}
I20260620 18:45:14.823040 21437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:14.822122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.130:60593)
0620 18:45:14.822761 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:14.822847 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":252,"threads_started":1}
I20260620 18:45:15.275672 21438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:15.274950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.131:41885)
0620 18:45:15.275415 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:15.275499 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":235,"threads_started":1}
I20260620 18:45:15.361325 21439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:15.360629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.129:45719)
0620 18:45:15.361072 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:15.361162 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":229,"threads_started":1}
I20260620 18:45:15.824636 21440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:15.823932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.130:33041)
0620 18:45:15.824381 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:15.824473 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":229,"threads_started":1}
I20260620 18:45:16.277446 21441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:16.276686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.131:50553)
0620 18:45:16.277179 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:16.277276 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":260,"threads_started":1}
I20260620 18:45:16.363129 21442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:16.362335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.129:56151)
0620 18:45:16.362815 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:16.362914 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":214,"threads_started":1}
I20260620 18:45:16.826465 21443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:16.825615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.130:59301)
0620 18:45:16.826129 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:16.826244 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":242,"threads_started":1}
I20260620 18:45:17.279259 21444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:17.278480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.131:33455)
0620 18:45:17.278947 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:17.279041 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":201,"threads_started":1}
I20260620 18:45:17.365067 21445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:17.364316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.129:38879)
0620 18:45:17.364810 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:17.364898 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":263,"threads_started":1}
I20260620 18:45:17.828251 21446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:17.827515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.130:57069)
0620 18:45:17.827991 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:17.828079 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":241,"threads_started":1}
I20260620 18:45:18.281302 21447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:18.280339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.131:37747)
0620 18:45:18.280918 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:18.281016 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":286,"threads_started":1}
I20260620 18:45:18.366935 21448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:18.366064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.129:44331)
0620 18:45:18.366625 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:18.366740 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":303,"threads_started":1}
I20260620 18:45:18.829975 21449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:18.829200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.130:57929)
0620 18:45:18.829694 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:18.829797 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":247,"threads_started":1}
I20260620 18:45:19.283129 21450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:19.282371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.131:49475)
0620 18:45:19.282847 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:19.282933 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":231,"threads_started":1}
I20260620 18:45:19.368584 21451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:19.367879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.129:54247)
0620 18:45:19.368332 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:19.368422 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":244,"threads_started":1}
I20260620 18:45:19.831977 21452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:19.831135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.130:45781)
0620 18:45:19.831673 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:19.831778 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":304,"threads_started":1}
I20260620 18:45:20.284773 21453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:20.284046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.131:40181)
0620 18:45:20.284528 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:20.284612 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":250,"threads_started":1}
I20260620 18:45:20.370391 21454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:20.369534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.129:37977)
0620 18:45:20.369995 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:20.370084 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":228,"threads_started":1}
I20260620 18:45:20.833721 21455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:20.832972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.130:36283)
0620 18:45:20.833442 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:20.833540 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":236,"threads_started":1}
I20260620 18:45:21.286527 21456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:21.285710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.131:60441)
0620 18:45:21.286145 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:21.286325 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":220,"threads_started":1}
I20260620 18:45:21.372313 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:21.371411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.129:35609)
0620 18:45:21.372075 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:21.372153 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":346,"threads_started":1}
I20260620 18:45:21.835541 21458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:21.834811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.130:49265)
0620 18:45:21.835290 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:21.835381 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":256,"threads_started":1}
I20260620 18:45:22.288314 21459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:22.287513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.131:41173)
0620 18:45:22.288033 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:22.288139 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":279,"threads_started":1}
I20260620 18:45:22.373891 21460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:22.373158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.129:43525)
0620 18:45:22.373605 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:22.373698 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":220,"threads_started":1}
I20260620 18:45:22.837376 21461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:22.836587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.130:39787)
0620 18:45:22.837084 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:22.837186 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":238,"threads_started":1}
I20260620 18:45:23.290056 21462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:23.289285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.131:38281)
0620 18:45:23.289743 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:23.289853 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":229,"threads_started":1}
I20260620 18:45:23.375514 21463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:23.374765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.129:47873)
0620 18:45:23.375224 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:23.375328 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":238,"threads_started":1}
I20260620 18:45:23.839087 21464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:23.838366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.130:53911)
0620 18:45:23.838814 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:23.838907 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":236,"threads_started":1}
I20260620 18:45:24.291700 21465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:24.290995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.131:33333)
0620 18:45:24.291433 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:24.291521 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":206,"threads_started":1}
I20260620 18:45:24.377399 21466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:24.376641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.129:40615)
0620 18:45:24.377093 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:24.377188 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":238,"threads_started":1}
I20260620 18:45:24.841179 21467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:24.840379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.130:50281)
0620 18:45:24.840890 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:24.841000 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":237,"threads_started":1}
I20260620 18:45:25.278025 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 20996
I20260620 18:45:25.293692 21469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:25.292898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36571 (local address 127.5.225.131:42045)
0620 18:45:25.293426 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:25.293515 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":279,"threads_started":1}
I20260620 18:45:25.297183 21468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:25.279312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38686 (local address 127.5.225.129:37807)
0620 18:45:25.280144 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:25.280165 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:25.280218 (+ 53us) server_negotiation.cc:407] Connection header received
0620 18:45:25.281950 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:25.281959 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:25.282488 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:25.282794 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:25.284261 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:25.285481 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:25.287003 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:25.287671 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:25.287834 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:25.290634 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:25.290642 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:25.290657 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:25.290695 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:25.293704 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:25.294593 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:25.294600 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:25.294606 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:25.294772 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:25.295176 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:25.295181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:25.295183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:25.295692 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:25.296028 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:25.296477 (+ 449us) server_negotiation.cc:299] Negotiation successful
0620 18:45:25.296902 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":458,"threads_started":1}
I20260620 18:45:25.298651 21153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:25.384471 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 20996
I20260620 18:45:25.404722 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 21161
I20260620 18:45:25.423102 21472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:25.405795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46046 (local address 127.5.225.130:40447)
0620 18:45:25.406412 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:25.406427 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:25.406464 (+ 37us) server_negotiation.cc:407] Connection header received
0620 18:45:25.408028 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:25.408038 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:25.408510 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:25.408821 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:25.410146 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:25.411457 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:25.412805 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:25.413503 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:25.413669 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:25.416573 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:25.416580 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:25.416596 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:25.416637 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:25.419601 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:25.420391 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:25.420398 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:25.420404 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:25.420592 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:25.421083 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:25.421087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:25.421089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:25.421636 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:25.421874 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:25.422361 (+ 487us) server_negotiation.cc:299] Negotiation successful
0620 18:45:25.422810 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":297,"threads_started":1}
I20260620 18:45:25.424655 21284 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:25.508294 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 21161
I20260620 18:45:25.526275 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 21292
I20260620 18:45:25.544831 21474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:25.527440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50464 (local address 127.5.225.131:38317)
0620 18:45:25.528020 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:25.528033 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:25.528075 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:45:25.529720 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:25.529730 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:25.530256 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:25.530626 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:25.532131 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:25.533336 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:25.534723 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:25.535413 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:25.535625 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:25.538364 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:25.538378 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:25.538399 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:25.538451 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:25.541503 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:25.542322 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:25.542329 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:25.542335 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:25.542526 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:25.542881 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:25.542884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:25.542886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:25.543430 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:25.543687 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:25.544066 (+ 379us) server_negotiation.cc:299] Negotiation successful
0620 18:45:25.544527 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":268,"threads_started":1}
I20260620 18:45:25.546355 21415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:25.629989 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 21292
2026-06-20T18:45:25Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24750 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:45:25 dist-test-slave-k6lg krb5kdc[21478](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:45:25 dist-test-slave-k6lg krb5kdc[21478](info): set up 2 sockets
Jun 20 18:45:25 dist-test-slave-k6lg krb5kdc[21478](info): commencing operation
krb5kdc: starting...
W20260620 18:45:26.754715 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.007s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:45:26 dist-test-slave-k6lg krb5kdc[21478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:45:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:45:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
I20260620 18:45:26.994335 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:45289
--webserver_interface=127.5.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:35541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:45289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:45:27.327276 21491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:27.327571 21491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:27.327731 21491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:27.335253 21491 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 18:45:27.335372 21491 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 18:45:27.335410 21491 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 18:45:27.335512 21491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:45:27.335613 21491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:27.335703 21491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:27.335773 21491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:45:27.335845 21491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:45:27.346033 21491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:35541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:45289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:45289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:45:27.347416 21491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:27.348809 21491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:27.361800 21496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:27.361881 21497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:27.362035 21491 server_base.cc:1034] running on GCE node
W20260620 18:45:27.361759 21499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:27.363061 21491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:27.364480 21491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:27.365702 21491 hybrid_clock.cc:648] HybridClock initialized: now 1781981127365616 us; error 59 us; skew 500 ppm
I20260620 18:45:27.366139 21491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:45:27 dist-test-slave-k6lg krb5kdc[21478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981127, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:45:27.369374 21491 init.cc:377] Logged in from keytab as kudu/127.5.225.190@KRBTEST.COM (short username kudu)
I20260620 18:45:27.371073 21491 webserver.cc:460] Webserver started at http://127.5.225.190:45995/ using document root <none> and password file <none>
I20260620 18:45:27.371789 21491 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:27.371882 21491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:27.372213 21491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:45:27.375401 21491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "0f9a644e0e0146b4a66ef256b1f7a049"
format_stamp: "Formatted at 2026-06-20 18:45:27 on dist-test-slave-k6lg"
server_key: "25f877fe481f72bdacc5bfa67cb852ac"
server_key_iv: "ca562b88a2576b4771e3a3fba1827c71"
server_key_version: "encryptionkey@0"
I20260620 18:45:27.376339 21491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "0f9a644e0e0146b4a66ef256b1f7a049"
format_stamp: "Formatted at 2026-06-20 18:45:27 on dist-test-slave-k6lg"
server_key: "25f877fe481f72bdacc5bfa67cb852ac"
server_key_iv: "ca562b88a2576b4771e3a3fba1827c71"
server_key_version: "encryptionkey@0"
I20260620 18:45:27.381640 21491 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 18:45:27.385259 21506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:27.386932 21491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:45:27.387135 21491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "0f9a644e0e0146b4a66ef256b1f7a049"
format_stamp: "Formatted at 2026-06-20 18:45:27 on dist-test-slave-k6lg"
server_key: "25f877fe481f72bdacc5bfa67cb852ac"
server_key_iv: "ca562b88a2576b4771e3a3fba1827c71"
server_key_version: "encryptionkey@0"
I20260620 18:45:27.387372 21491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:27.412690 21491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:27.416894 21491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:27.417172 21491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:27.436825 21491 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:45289
I20260620 18:45:27.436851 21557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:45289 every 8 connection(s)
I20260620 18:45:27.438283 21491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:45:27.442226 21558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:45:27.447758 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 21491
I20260620 18:45:27.448029 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:45:27.448658 6022 external_mini_cluster.cc:1428] Setting key 0fd25dd46235589786ef958c56927886
I20260620 18:45:27.454051 21558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049: Bootstrap starting.
I20260620 18:45:27.457893 21558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049: Neither blocks nor log segments found. Creating new log.
I20260620 18:45:27.459187 21558 log.cc:824] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049: Log is configured to *not* fsync() on all Append() calls
I20260620 18:45:27.462103 21558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049: No bootstrap required, opened a new log
Jun 20 18:45:27 dist-test-slave-k6lg krb5kdc[21478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:45:27.470430 21558 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f9a644e0e0146b4a66ef256b1f7a049" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 45289 } }
I20260620 18:45:27.470883 21558 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:45:27.471027 21558 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f9a644e0e0146b4a66ef256b1f7a049, State: Initialized, Role: FOLLOWER
I20260620 18:45:27.471767 21558 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f9a644e0e0146b4a66ef256b1f7a049" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 45289 } }
I20260620 18:45:27.472077 21558 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:45:27.472239 21558 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:45:27.472455 21558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:45:27.473412 21561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:27.451153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38518 (local address 127.5.225.190:45289)
0620 18:45:27.451972 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:27.451990 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:27.452038 (+ 48us) server_negotiation.cc:407] Connection header received
0620 18:45:27.454410 (+ 2372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:27.454439 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:27.454950 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:27.455381 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:27.456755 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:27.458819 (+ 2064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:27.460291 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:27.461001 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:27.461206 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:27.465445 (+ 4239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:27.465453 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:27.465469 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:27.465514 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:27.468855 (+ 3341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:27.469643 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:27.469650 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:27.469657 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:27.469855 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:27.470296 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:27.470301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:27.470306 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:27.470918 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:27.471247 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:27.471704 (+ 457us) server_negotiation.cc:299] Negotiation successful
0620 18:45:27.472115 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":369,"threads_started":1}
I20260620 18:45:27.475636 21558 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f9a644e0e0146b4a66ef256b1f7a049" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 45289 } }
I20260620 18:45:27.476157 21558 leader_election.cc:302] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f9a644e0e0146b4a66ef256b1f7a049; no voters:
I20260620 18:45:27.476857 21558 leader_election.cc:288] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:45:27.476985 21563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:45:27.477694 21563 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [term 1 LEADER]: Becoming Leader. State: Replica: 0f9a644e0e0146b4a66ef256b1f7a049, State: Running, Role: LEADER
I20260620 18:45:27.478334 21563 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f9a644e0e0146b4a66ef256b1f7a049" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 45289 } }
I20260620 18:45:27.479043 21558 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:45:27.481283 21564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f9a644e0e0146b4a66ef256b1f7a049" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f9a644e0e0146b4a66ef256b1f7a049" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 45289 } } }
I20260620 18:45:27.481740 21564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [sys.catalog]: This master's current role is: LEADER
I20260620 18:45:27.484195 21565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f9a644e0e0146b4a66ef256b1f7a049. Latest consensus state: current_term: 1 leader_uuid: "0f9a644e0e0146b4a66ef256b1f7a049" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f9a644e0e0146b4a66ef256b1f7a049" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 45289 } } }
I20260620 18:45:27.484642 21565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [sys.catalog]: This master's current role is: LEADER
I20260620 18:45:27.487524 21572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:45:27.492400 21572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:45:27.500604 21572 catalog_manager.cc:1347] Generated new cluster ID: ad104ace8f224e259c912cbb453d1f6b
I20260620 18:45:27.500731 21572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:45:27.523967 21572 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:45:27.525334 21572 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:45:27.534960 21572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049: Generated new TSK 0
I20260620 18:45:27.536095 21572 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab.
Entry for principal kudu/127.5.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab.
I20260620 18:45:27.599050 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.129:0
--local_ip_for_outbound_sockets=127.5.225.129
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:45289
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:35541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:45:27.928843 21584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:27.929124 21584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:27.929287 21584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:27.936538 21584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:27.936662 21584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:27.936852 21584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.129
I20260620 18:45:27.948236 21584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:35541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:45289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.129
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:45:27.949556 21584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:27.950975 21584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:27.964352 21590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:27.964331 21589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:27.964588 21592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:27.965130 21584 server_base.cc:1034] running on GCE node
I20260620 18:45:27.965732 21584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:27.966673 21584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:27.967900 21584 hybrid_clock.cc:648] HybridClock initialized: now 1781981127967861 us; error 52 us; skew 500 ppm
I20260620 18:45:27.968231 21584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:45:27 dist-test-slave-k6lg krb5kdc[21478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981127, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:45:27.971412 21584 init.cc:377] Logged in from keytab as kudu/127.5.225.129@KRBTEST.COM (short username kudu)
I20260620 18:45:27.972911 21584 webserver.cc:460] Webserver started at http://127.5.225.129:34483/ using document root <none> and password file <none>
I20260620 18:45:27.973600 21584 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:27.973744 21584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:27.974071 21584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:45:27.977042 21584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/data/instance:
uuid: "e1f9f638263945d7b85ff6c2c237c2f1"
format_stamp: "Formatted at 2026-06-20 18:45:27 on dist-test-slave-k6lg"
server_key: "f0a4c2e6ed04336be52dda6992a19228"
server_key_iv: "ec7e732668ae09a39358427b02d70eb5"
server_key_version: "encryptionkey@0"
I20260620 18:45:27.978014 21584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance:
uuid: "e1f9f638263945d7b85ff6c2c237c2f1"
format_stamp: "Formatted at 2026-06-20 18:45:27 on dist-test-slave-k6lg"
server_key: "f0a4c2e6ed04336be52dda6992a19228"
server_key_iv: "ec7e732668ae09a39358427b02d70eb5"
server_key_version: "encryptionkey@0"
I20260620 18:45:27.983577 21584 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:45:27.987438 21599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:27.988811 21584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 18:45:27.988986 21584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/wal
uuid: "e1f9f638263945d7b85ff6c2c237c2f1"
format_stamp: "Formatted at 2026-06-20 18:45:27 on dist-test-slave-k6lg"
server_key: "f0a4c2e6ed04336be52dda6992a19228"
server_key_iv: "ec7e732668ae09a39358427b02d70eb5"
server_key_version: "encryptionkey@0"
I20260620 18:45:27.989197 21584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:28.010040 21584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:28.014526 21584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:28.014827 21584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:28.015811 21584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:45:28.017138 21584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:45:28.017215 21584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:28.017390 21584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:45:28.017452 21584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:28.050061 21584 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.129:41609
I20260620 18:45:28.050143 21711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.129:41609 every 8 connection(s)
I20260620 18:45:28.051385 21584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
I20260620 18:45:28.052029 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 21584
I20260620 18:45:28.052280 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance
I20260620 18:45:28.053087 6022 external_mini_cluster.cc:1428] Setting key da8ee8ccc72e1941cf07f043b88bb802
Jun 20 18:45:28 dist-test-slave-k6lg krb5kdc[21478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981127, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:45:28.074311 21714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:28.054382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:56555 (local address 127.5.225.190:45289)
0620 18:45:28.054879 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:28.054886 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:28.056140 (+ 1254us) server_negotiation.cc:407] Connection header received
0620 18:45:28.058050 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:28.058055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:28.058140 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:28.058322 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:28.059848 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:28.060600 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:28.062690 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:28.063094 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:28.063201 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:28.067401 (+ 4200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:28.067411 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:28.067419 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:28.067463 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:28.071042 (+ 3579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:28.071764 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:28.071772 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:28.071778 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:28.071934 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:28.072367 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:28.072375 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:28.072381 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:28.072759 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:28.072930 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:28.073676 (+ 746us) server_negotiation.cc:299] Negotiation successful
0620 18:45:28.073899 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":229,"threads_started":1}
I20260620 18:45:28.075397 21715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:28.055110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.129:56555)
0620 18:45:28.055989 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:28.056035 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:28.057777 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:28.058555 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:28.058564 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:28.059111 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:28.059667 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:28.059684 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:28.060745 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:28.060761 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:28.062531 (+ 1770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:28.062550 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:28.063331 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:28.063341 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:28.063676 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:28.064400 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:28.064439 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:28.067171 (+ 2732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:45:28.071174 (+ 4003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:28.071181 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:28.071193 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:28.071623 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:28.072058 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:28.072061 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:28.072063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:28.072234 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:28.072936 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:45:28.072942 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:45:28.073261 (+ 319us) client_negotiation.cc:769] Sending connection context
0620 18:45:28.073570 (+ 309us) client_negotiation.cc:240] Negotiation successful
0620 18:45:28.074108 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":367,"threads_started":1}
I20260620 18:45:28.077073 21712 heartbeater.cc:347] Connected to a master server at 127.5.225.190:45289
I20260620 18:45:28.077385 21712 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.5.225.130@KRBTEST.COM; defaulting to no policy
I20260620 18:45:28.078279 21712 heartbeater.cc:511] Master 127.5.225.190:45289 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.130@KRBTEST.COM" created.
I20260620 18:45:28.080749 21523 ts_manager.cc:194] Registered new tserver with Master: e1f9f638263945d7b85ff6c2c237c2f1 (127.5.225.129:41609)
I20260620 18:45:28.082314 21523 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.129@KRBTEST.COM'} at 127.5.225.129:56555
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab.
Entry for principal kudu/127.5.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab.
I20260620 18:45:28.109556 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.130:0
--local_ip_for_outbound_sockets=127.5.225.130
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:45289
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:35541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:45:28.432593 21718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:28.432893 21718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:28.433079 21718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:28.440464 21718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:28.440623 21718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:28.440796 21718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.130
I20260620 18:45:28.450695 21718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:35541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:45289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.130
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:45:28.452031 21718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:28.453311 21718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:28.466411 21723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:28.466444 21726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:28.466475 21724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:28.467341 21718 server_base.cc:1034] running on GCE node
I20260620 18:45:28.467896 21718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:28.468737 21718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:28.469959 21718 hybrid_clock.cc:648] HybridClock initialized: now 1781981128469917 us; error 35 us; skew 500 ppm
I20260620 18:45:28.470321 21718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:45:28 dist-test-slave-k6lg krb5kdc[21478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981128, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:45:28.473340 21718 init.cc:377] Logged in from keytab as kudu/127.5.225.130@KRBTEST.COM (short username kudu)
I20260620 18:45:28.474828 21718 webserver.cc:460] Webserver started at http://127.5.225.130:43493/ using document root <none> and password file <none>
I20260620 18:45:28.475487 21718 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:28.475621 21718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:28.475953 21718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:45:28.478852 21718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/data/instance:
uuid: "7d8f23504b8141be866b091a88bd6ace"
format_stamp: "Formatted at 2026-06-20 18:45:28 on dist-test-slave-k6lg"
server_key: "1431d5002b2e2fba1bb57522ac2fef17"
server_key_iv: "e6d97a8d9a471bb66ed86867698a539d"
server_key_version: "encryptionkey@0"
I20260620 18:45:28.479633 21718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance:
uuid: "7d8f23504b8141be866b091a88bd6ace"
format_stamp: "Formatted at 2026-06-20 18:45:28 on dist-test-slave-k6lg"
server_key: "1431d5002b2e2fba1bb57522ac2fef17"
server_key_iv: "e6d97a8d9a471bb66ed86867698a539d"
server_key_version: "encryptionkey@0"
I20260620 18:45:28.484845 21718 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 18:45:28.488803 21733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:28.490314 21718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:45:28.490500 21718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/wal
uuid: "7d8f23504b8141be866b091a88bd6ace"
format_stamp: "Formatted at 2026-06-20 18:45:28 on dist-test-slave-k6lg"
server_key: "1431d5002b2e2fba1bb57522ac2fef17"
server_key_iv: "e6d97a8d9a471bb66ed86867698a539d"
server_key_version: "encryptionkey@0"
I20260620 18:45:28.490700 21718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:28.518538 21718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:28.522702 21718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:28.523129 21718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:28.524165 21718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:45:28.525513 21718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:45:28.525614 21718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:28.525763 21718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:45:28.525842 21718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:28.559044 21718 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.130:34311
I20260620 18:45:28.559084 21845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.130:34311 every 8 connection(s)
I20260620 18:45:28.560333 21718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
I20260620 18:45:28.563288 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 21718
I20260620 18:45:28.563496 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance
I20260620 18:45:28.563961 6022 external_mini_cluster.cc:1428] Setting key 3e1bff2a01040590319f5f088605c53d
Jun 20 18:45:28 dist-test-slave-k6lg krb5kdc[21478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981128, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:45:28.582597 21714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:28.562764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:34505 (local address 127.5.225.190:45289)
0620 18:45:28.562946 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:28.562951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:28.564237 (+ 1286us) server_negotiation.cc:407] Connection header received
0620 18:45:28.566420 (+ 2183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:28.566426 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:28.566510 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:28.566622 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:28.568366 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:28.569556 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:28.571815 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:28.572560 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:28.572698 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:28.576761 (+ 4063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:28.576766 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:28.576769 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:28.576800 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:28.579515 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:28.580288 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:28.580293 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:28.580299 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:28.580434 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:28.580876 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:28.580881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:28.580884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:28.581146 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:28.581307 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:28.581987 (+ 680us) server_negotiation.cc:299] Negotiation successful
0620 18:45:28.582251 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:45:28.583644 21848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:28.563228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.130:34505)
0620 18:45:28.564053 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:28.564105 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:28.566086 (+ 1981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:28.566911 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:28.566920 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:28.567609 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:28.568184 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:28.568201 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:28.569733 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:28.569741 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:28.571660 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:28.571676 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:28.572767 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:28.572775 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:28.573068 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:28.573856 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:28.573887 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:28.576538 (+ 2651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:45:28.579649 (+ 3111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:28.579657 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:28.579673 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:28.580166 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:28.580554 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:28.580558 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:28.580562 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:28.580742 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:28.581278 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:45:28.581285 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:45:28.581614 (+ 329us) client_negotiation.cc:769] Sending connection context
0620 18:45:28.581887 (+ 273us) client_negotiation.cc:240] Negotiation successful
0620 18:45:28.582437 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":367,"threads_started":1}
I20260620 18:45:28.585261 21846 heartbeater.cc:347] Connected to a master server at 127.5.225.190:45289
I20260620 18:45:28.585597 21846 heartbeater.cc:464] Registering TS with master...
I20260620 18:45:28.586364 21846 heartbeater.cc:511] Master 127.5.225.190:45289 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.225.131@KRBTEST.COM; defaulting to no policy
I20260620 18:45:28.587759 21523 ts_manager.cc:194] Registered new tserver with Master: 7d8f23504b8141be866b091a88bd6ace (127.5.225.130:34311)
I20260620 18:45:28.588619 21523 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.130@KRBTEST.COM'} at 127.5.225.130:34505
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab.
Entry for principal kudu/127.5.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab.
I20260620 18:45:28.621417 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.131:0
--local_ip_for_outbound_sockets=127.5.225.131
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:45289
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:35541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:45:28.950261 21851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:28.950546 21851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:28.950644 21851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:28.957844 21851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:28.957950 21851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:28.958065 21851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.131
I20260620 18:45:28.967940 21851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:35541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:45289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.131
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:45:28.969163 21851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:28.970533 21851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:28.983624 21856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:28.983671 21857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:28.983914 21859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:28.984635 21851 server_base.cc:1034] running on GCE node
I20260620 18:45:28.985399 21851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:28.986392 21851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:28.987597 21851 hybrid_clock.cc:648] HybridClock initialized: now 1781981128987557 us; error 30 us; skew 500 ppm
I20260620 18:45:28.987957 21851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:45:28 dist-test-slave-k6lg krb5kdc[21478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981128, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:45:28.991191 21851 init.cc:377] Logged in from keytab as kudu/127.5.225.131@KRBTEST.COM (short username kudu)
I20260620 18:45:28.993059 21851 webserver.cc:460] Webserver started at http://127.5.225.131:36111/ using document root <none> and password file <none>
I20260620 18:45:28.993764 21851 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:28.993930 21851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:28.994336 21851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:45:28.997365 21851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/data/instance:
uuid: "6f8524c2278f4cd29d9f88ba41228cc6"
format_stamp: "Formatted at 2026-06-20 18:45:28 on dist-test-slave-k6lg"
server_key: "ef8f8c8c7e070ddf046e4f98e6e8321c"
server_key_iv: "9f0c21d2af9a17cd04dda8d1b6efe893"
server_key_version: "encryptionkey@0"
I20260620 18:45:28.998124 21851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance:
uuid: "6f8524c2278f4cd29d9f88ba41228cc6"
format_stamp: "Formatted at 2026-06-20 18:45:28 on dist-test-slave-k6lg"
server_key: "ef8f8c8c7e070ddf046e4f98e6e8321c"
server_key_iv: "9f0c21d2af9a17cd04dda8d1b6efe893"
server_key_version: "encryptionkey@0"
I20260620 18:45:29.003700 21851 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:45:29.007329 21866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:29.008987 21851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:45:29.009282 21851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/wal
uuid: "6f8524c2278f4cd29d9f88ba41228cc6"
format_stamp: "Formatted at 2026-06-20 18:45:28 on dist-test-slave-k6lg"
server_key: "ef8f8c8c7e070ddf046e4f98e6e8321c"
server_key_iv: "9f0c21d2af9a17cd04dda8d1b6efe893"
server_key_version: "encryptionkey@0"
I20260620 18:45:29.009500 21851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:29.024431 21851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:29.028437 21851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:29.028733 21851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:29.029726 21851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:45:29.031066 21851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:45:29.031168 21851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:29.031311 21851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:45:29.031390 21851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:29.065812 21851 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.131:43087
I20260620 18:45:29.065855 21978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.131:43087 every 8 connection(s)
I20260620 18:45:29.067148 21851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
I20260620 18:45:29.074596 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 21851
I20260620 18:45:29.074831 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance
I20260620 18:45:29.075397 6022 external_mini_cluster.cc:1428] Setting key c5a5a6a6542d27f52e4465b2ccc21836
Jun 20 18:45:29 dist-test-slave-k6lg krb5kdc[21478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981128, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:45:29.085538 21712 heartbeater.cc:503] Master 127.5.225.190:45289 was elected leader, sending a full tablet report...
I20260620 18:45:29.088776 21714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:29.069655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:47563 (local address 127.5.225.190:45289)
0620 18:45:29.069915 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:29.069922 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:29.071145 (+ 1223us) server_negotiation.cc:407] Connection header received
0620 18:45:29.072910 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:29.072915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:29.073000 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:29.073091 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:29.074822 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:29.075953 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:29.078259 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:29.078844 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:29.079001 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:29.082682 (+ 3681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:29.082687 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:29.082690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:29.082723 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:29.085546 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:29.086411 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:29.086423 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:29.086426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:29.086575 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:29.087027 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:29.087032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:29.087035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:29.087308 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:29.087468 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:29.088174 (+ 706us) server_negotiation.cc:299] Negotiation successful
0620 18:45:29.088402 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 18:45:29.090013 21981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:29.070260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.131:47563)
0620 18:45:29.070979 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:29.071018 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:29.072638 (+ 1620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:29.073297 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:29.073305 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:29.073832 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:29.074576 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:29.074595 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:29.076137 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:29.076145 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:29.078031 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:29.078045 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:29.079115 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:29.079121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:29.079331 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:29.080067 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:29.080116 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:29.082433 (+ 2317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:45:29.085696 (+ 3263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:29.085705 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:29.085723 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:29.086276 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:29.086695 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:29.086701 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:29.086704 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:29.086899 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:29.087442 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:45:29.087450 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:45:29.087804 (+ 354us) client_negotiation.cc:769] Sending connection context
0620 18:45:29.088076 (+ 272us) client_negotiation.cc:240] Negotiation successful
0620 18:45:29.088633 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":276,"threads_started":1}
I20260620 18:45:29.091476 21979 heartbeater.cc:347] Connected to a master server at 127.5.225.190:45289
I20260620 18:45:29.091804 21979 heartbeater.cc:464] Registering TS with master...
I20260620 18:45:29.092628 21979 heartbeater.cc:511] Master 127.5.225.190:45289 requested a full tablet report, sending...
I20260620 18:45:29.094048 21523 ts_manager.cc:194] Registered new tserver with Master: 6f8524c2278f4cd29d9f88ba41228cc6 (127.5.225.131:43087)
I20260620 18:45:29.094920 21523 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.131@KRBTEST.COM'} at 127.5.225.131:47563
I20260620 18:45:29.104079 6022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:45:29.121548 21714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:29.108576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38522 (local address 127.5.225.190:45289)
0620 18:45:29.108758 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:29.108766 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:29.109209 (+ 443us) server_negotiation.cc:407] Connection header received
0620 18:45:29.109428 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:29.109432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:29.109495 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:29.109605 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:29.111122 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:29.111853 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:29.113315 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:29.113824 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:29.113996 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:29.116447 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:29.116452 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:29.116455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:29.116486 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:29.118942 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:29.119602 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:29.119606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:29.119609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:29.119735 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:29.120119 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:29.120127 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:29.120131 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:29.120443 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:29.120703 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:29.120995 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 18:45:29.121214 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 18:45:29.125664 21523 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:45:29.132507 21523 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:45:29.155174 21992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:29.144147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39456 (local address 127.5.225.130:34311)
0620 18:45:29.144715 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:29.144725 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:29.145981 (+ 1256us) server_negotiation.cc:407] Connection header received
0620 18:45:29.146346 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:29.146354 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:29.146614 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:29.146780 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:45:29.147855 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:29.149398 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:29.152197 (+ 2799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:29.153655 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:29.153774 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:29.154002 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:29.154682 (+ 680us) server_negotiation.cc:299] Negotiation successful
0620 18:45:29.154863 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":293,"threads_started":1}
I20260620 18:45:29.156052 21991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:29.144018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.130:34311 (local address 127.0.0.1:39456)
0620 18:45:29.145871 (+ 1853us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:29.145887 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:29.146092 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:29.147127 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:29.147136 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:29.147166 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:45:29.147632 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:29.147650 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:29.149573 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:29.149578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:29.152023 (+ 2445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:29.152038 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:29.154206 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:29.154212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:29.154421 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:29.154470 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 18:45:29.155473 (+ 1003us) client_negotiation.cc:240] Negotiation successful
0620 18:45:29.155681 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":863,"thread_start_us":717,"threads_started":1}
I20260620 18:45:29.157600 21993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:29.144120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:41609 (local address 127.0.0.1:33684)
0620 18:45:29.145503 (+ 1383us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:29.145554 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:29.145721 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:29.147247 (+ 1526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:29.147253 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:29.147281 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:45:29.147718 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:29.147731 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:29.150619 (+ 2888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:29.150625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:29.153665 (+ 3040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:29.153680 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:29.156821 (+ 3141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:29.156828 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:29.157009 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:29.157047 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 18:45:29.157142 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 18:45:29.157269 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1212,"thread_start_us":1079,"threads_started":1}
I20260620 18:45:29.158480 21994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:29.144689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33684 (local address 127.5.225.129:41609)
0620 18:45:29.146449 (+ 1760us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:29.146457 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:29.146483 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:45:29.146565 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:29.146572 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:29.146780 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:29.146984 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:45:29.148876 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:29.150452 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:29.153865 (+ 3413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:29.156599 (+ 2734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:29.156761 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:29.157018 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:29.157971 (+ 953us) server_negotiation.cc:299] Negotiation successful
0620 18:45:29.158124 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1593,"thread_start_us":1447,"threads_started":1}
I20260620 18:45:29.161110 21781 tablet_service.cc:1467] Processing CreateTablet for tablet 5d8494f33d7e46b6b106336f5a9242d1 (DEFAULT_TABLE table=test-table [id=457ed6eaa9614e1f8758d5897ea01f21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:45:29.162576 21781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d8494f33d7e46b6b106336f5a9242d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:45:29.162616 21647 tablet_service.cc:1467] Processing CreateTablet for tablet 5d8494f33d7e46b6b106336f5a9242d1 (DEFAULT_TABLE table=test-table [id=457ed6eaa9614e1f8758d5897ea01f21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:45:29.163863 21647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d8494f33d7e46b6b106336f5a9242d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:45:29.165266 21991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:29.145637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.131:43087 (local address 127.0.0.1:51268)
0620 18:45:29.156299 (+ 10662us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:29.156317 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:29.156430 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:29.158417 (+ 1987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:29.158423 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:29.158452 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:45:29.158643 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:29.158656 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:29.160246 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:29.160251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:29.162697 (+ 2446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:29.162712 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:29.164486 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:29.164492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:29.164663 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:29.164706 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 18:45:29.164811 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 18:45:29.164965 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10559,"thread_start_us":11014,"threads_started":1}
I20260620 18:45:29.165393 21996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:29.146330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51268 (local address 127.5.225.131:43087)
0620 18:45:29.157709 (+ 11379us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:29.157721 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:29.157764 (+ 43us) server_negotiation.cc:407] Connection header received
0620 18:45:29.157863 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:29.157873 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:29.158215 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:29.158428 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:45:29.158862 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:29.160028 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:29.162903 (+ 2875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:29.164310 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:29.164466 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:29.164711 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:29.164870 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 18:45:29.165045 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11185,"thread_start_us":11024,"threads_started":1}
I20260620 18:45:29.169229 21914 tablet_service.cc:1467] Processing CreateTablet for tablet 5d8494f33d7e46b6b106336f5a9242d1 (DEFAULT_TABLE table=test-table [id=457ed6eaa9614e1f8758d5897ea01f21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:45:29.170701 21914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d8494f33d7e46b6b106336f5a9242d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:45:29.175601 21997 tablet_bootstrap.cc:492] T 5d8494f33d7e46b6b106336f5a9242d1 P 7d8f23504b8141be866b091a88bd6ace: Bootstrap starting.
I20260620 18:45:29.179147 21997 tablet_bootstrap.cc:654] T 5d8494f33d7e46b6b106336f5a9242d1 P 7d8f23504b8141be866b091a88bd6ace: Neither blocks nor log segments found. Creating new log.
I20260620 18:45:29.180409 21997 log.cc:824] T 5d8494f33d7e46b6b106336f5a9242d1 P 7d8f23504b8141be866b091a88bd6ace: Log is configured to *not* fsync() on all Append() calls
I20260620 18:45:29.187706 21998 tablet_bootstrap.cc:492] T 5d8494f33d7e46b6b106336f5a9242d1 P e1f9f638263945d7b85ff6c2c237c2f1: Bootstrap starting.
I20260620 18:45:29.188908 22000 tablet_bootstrap.cc:492] T 5d8494f33d7e46b6b106336f5a9242d1 P 6f8524c2278f4cd29d9f88ba41228cc6: Bootstrap starting.
I20260620 18:45:29.190829 21997 tablet_bootstrap.cc:492] T 5d8494f33d7e46b6b106336f5a9242d1 P 7d8f23504b8141be866b091a88bd6ace: No bootstrap required, opened a new log
I20260620 18:45:29.191056 21998 tablet_bootstrap.cc:654] T 5d8494f33d7e46b6b106336f5a9242d1 P e1f9f638263945d7b85ff6c2c237c2f1: Neither blocks nor log segments found. Creating new log.
I20260620 18:45:29.191120 21997 ts_tablet_manager.cc:1397] T 5d8494f33d7e46b6b106336f5a9242d1 P 7d8f23504b8141be866b091a88bd6ace: Time spent bootstrapping tablet: real 0.016s user 0.014s sys 0.000s
I20260620 18:45:29.192176 21998 log.cc:824] T 5d8494f33d7e46b6b106336f5a9242d1 P e1f9f638263945d7b85ff6c2c237c2f1: Log is configured to *not* fsync() on all Append() calls
I20260620 18:45:29.193856 22000 tablet_bootstrap.cc:654] T 5d8494f33d7e46b6b106336f5a9242d1 P 6f8524c2278f4cd29d9f88ba41228cc6: Neither blocks nor log segments found. Creating new log.
I20260620 18:45:29.195168 22000 log.cc:824] T 5d8494f33d7e46b6b106336f5a9242d1 P 6f8524c2278f4cd29d9f88ba41228cc6: Log is configured to *not* fsync() on all Append() calls
I20260620 18:45:29.195389 21998 tablet_bootstrap.cc:492] T 5d8494f33d7e46b6b106336f5a9242d1 P e1f9f638263945d7b85ff6c2c237c2f1: No bootstrap required, opened a new log
I20260620 18:45:29.195681 21998 ts_tablet_manager.cc:1397] T 5d8494f33d7e46b6b106336f5a9242d1 P e1f9f638263945d7b85ff6c2c237c2f1: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 18:45:29.198268 22000 tablet_bootstrap.cc:492] T 5d8494f33d7e46b6b106336f5a9242d1 P 6f8524c2278f4cd29d9f88ba41228cc6: No bootstrap required, opened a new log
I20260620 18:45:29.198531 22000 ts_tablet_manager.cc:1397] T 5d8494f33d7e46b6b106336f5a9242d1 P 6f8524c2278f4cd29d9f88ba41228cc6: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260620 18:45:29.201463 21997 raft_consensus.cc:348] T 5d8494f33d7e46b6b106336f5a9242d1 P 7d8f23504b8141be866b091a88bd6ace [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d8f23504b8141be866b091a88bd6ace" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 34311 } } peers { permanent_uuid: "e1f9f638263945d7b85ff6c2c237c2f1" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 41609 } } peers { permanent_uuid: "6f8524c2278f4cd29d9f88ba41228cc6" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 43087 } }
I20260620 18:45:29.201882 21997 raft_consensus.cc:374] T 5d8494f33d7e46b6b106336f5a9242d1 P 7d8f23504b8141be866b091a88bd6ace [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:45:29.201975 21997 raft_consensus.cc:729] T 5d8494f33d7e46b6b106336f5a9242d1 P 7d8f23504b8141be866b091a88bd6ace [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d8f23504b8141be866b091a88bd6ace, State: Initialized, Role: FOLLOWER
I20260620 18:45:29.202617 21997 consensus_queue.cc:260] T 5d8494f33d7e46b6b106336f5a9242d1 P 7d8f23504b8141be866b091a88bd6ace [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d8f23504b8141be866b091a88bd6ace" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 34311 } } peers { permanent_uuid: "e1f9f638263945d7b85ff6c2c237c2f1" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 41609 } } peers { permanent_uuid: "6f8524c2278f4cd29d9f88ba41228cc6" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 43087 } }
I20260620 18:45:29.203120 21998 raft_consensus.cc:348] T 5d8494f33d7e46b6b106336f5a9242d1 P e1f9f638263945d7b85ff6c2c237c2f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d8f23504b8141be866b091a88bd6ace" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 34311 } } peers { permanent_uuid: "e1f9f638263945d7b85ff6c2c237c2f1" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 41609 } } peers { permanent_uuid: "6f8524c2278f4cd29d9f88ba41228cc6" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 43087 } }
I20260620 18:45:29.203569 21998 raft_consensus.cc:374] T 5d8494f33d7e46b6b106336f5a9242d1 P e1f9f638263945d7b85ff6c2c237c2f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:45:29.203672 21998 raft_consensus.cc:729] T 5d8494f33d7e46b6b106336f5a9242d1 P e1f9f638263945d7b85ff6c2c237c2f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1f9f638263945d7b85ff6c2c237c2f1, State: Initialized, Role: FOLLOWER
I20260620 18:45:29.203882 21846 heartbeater.cc:503] Master 127.5.225.190:45289 was elected leader, sending a full tablet report...
I20260620 18:45:29.204236 21998 consensus_queue.cc:260] T 5d8494f33d7e46b6b106336f5a9242d1 P e1f9f638263945d7b85ff6c2c237c2f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d8f23504b8141be866b091a88bd6ace" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 34311 } } peers { permanent_uuid: "e1f9f638263945d7b85ff6c2c237c2f1" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 41609 } } peers { permanent_uuid: "6f8524c2278f4cd29d9f88ba41228cc6" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 43087 } }
I20260620 18:45:29.204504 21997 ts_tablet_manager.cc:1428] T 5d8494f33d7e46b6b106336f5a9242d1 P 7d8f23504b8141be866b091a88bd6ace: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260620 18:45:29.206065 21998 ts_tablet_manager.cc:1428] T 5d8494f33d7e46b6b106336f5a9242d1 P e1f9f638263945d7b85ff6c2c237c2f1: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260620 18:45:29.208798 22000 raft_consensus.cc:348] T 5d8494f33d7e46b6b106336f5a9242d1 P 6f8524c2278f4cd29d9f88ba41228cc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d8f23504b8141be866b091a88bd6ace" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 34311 } } peers { permanent_uuid: "e1f9f638263945d7b85ff6c2c237c2f1" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 41609 } } peers { permanent_uuid: "6f8524c2278f4cd29d9f88ba41228cc6" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 43087 } }
I20260620 18:45:29.209241 22000 raft_consensus.cc:374] T 5d8494f33d7e46b6b106336f5a9242d1 P 6f8524c2278f4cd29d9f88ba41228cc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:45:29.209357 22000 raft_consensus.cc:729] T 5d8494f33d7e46b6b106336f5a9242d1 P 6f8524c2278f4cd29d9f88ba41228cc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f8524c2278f4cd29d9f88ba41228cc6, State: Initialized, Role: FOLLOWER
I20260620 18:45:29.209975 22000 consensus_queue.cc:260] T 5d8494f33d7e46b6b106336f5a9242d1 P 6f8524c2278f4cd29d9f88ba41228cc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d8f23504b8141be866b091a88bd6ace" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 34311 } } peers { permanent_uuid: "e1f9f638263945d7b85ff6c2c237c2f1" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 41609 } } peers { permanent_uuid: "6f8524c2278f4cd29d9f88ba41228cc6" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 43087 } }
I20260620 18:45:29.212127 21979 heartbeater.cc:503] Master 127.5.225.190:45289 was elected leader, sending a full tablet report...
I20260620 18:45:29.212469 22000 ts_tablet_manager.cc:1428] T 5d8494f33d7e46b6b106336f5a9242d1 P 6f8524c2278f4cd29d9f88ba41228cc6: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260620 18:45:29.280648 22005 raft_consensus.cc:482] T 5d8494f33d7e46b6b106336f5a9242d1 P 6f8524c2278f4cd29d9f88ba41228cc6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:45:29.280972 22005 raft_consensus.cc:504] T 5d8494f33d7e46b6b106336f5a9242d1 P 6f8524c2278f4cd29d9f88ba41228cc6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d8f23504b8141be866b091a88bd6ace" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 34311 } } peers { permanent_uuid: "e1f9f638263945d7b85ff6c2c237c2f1" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 41609 } } peers { permanent_uuid: "6f8524c2278f4cd29d9f88ba41228cc6" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 43087 } }
I20260620 18:45:29.282789 22005 leader_election.cc:288] T 5d8494f33d7e46b6b106336f5a9242d1 P 6f8524c2278f4cd29d9f88ba41228cc6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d8f23504b8141be866b091a88bd6ace (127.5.225.130:34311), e1f9f638263945d7b85ff6c2c237c2f1 (127.5.225.129:41609)
I20260620 18:45:29.289405 21992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:29.283551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:58343 (local address 127.5.225.130:34311)
0620 18:45:29.283781 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:29.283786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:29.283808 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:45:29.283891 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:29.283899 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:29.283976 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:29.284096 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:45:29.284592 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:29.285576 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:29.287481 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:29.288600 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:29.288709 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:29.288760 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:29.289120 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 18:45:29.289214 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 18:45:29.289562 21981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:29.283172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.130:34311 (local address 127.5.225.131:58343)
0620 18:45:29.283399 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:29.283421 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:29.283552 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:29.284161 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:29.284175 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:29.284198 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:45:29.284442 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:29.284452 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:29.285729 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:29.285736 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:29.287325 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:29.287337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:29.288745 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:29.288752 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:29.288943 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:29.288983 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 18:45:29.289079 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 18:45:29.289217 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":38,"spinlock_wait_cycles":18304}
I20260620 18:45:29.290444 21994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:29.283288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:57931 (local address 127.5.225.129:41609)
0620 18:45:29.283491 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:29.283496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:29.284176 (+ 680us) server_negotiation.cc:407] Connection header received
0620 18:45:29.284454 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:29.284461 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:29.284547 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:29.284630 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:45:29.285288 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:29.286313 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:29.288337 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:29.289484 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:29.289648 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:29.289720 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:29.290113 (+ 393us) server_negotiation.cc:299] Negotiation successful
0620 18:45:29.290265 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:45:29.290573 22006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:29.283172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:41609 (local address 127.5.225.131:57931)
0620 18:45:29.284054 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:29.284074 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:29.284292 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:29.284669 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:29.284676 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:29.284704 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:45:29.285083 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:29.285095 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:29.286479 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:29.286483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:29.288174 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:29.288190 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:29.289806 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:29.289813 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:29.289955 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:29.289979 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 18:45:29.290064 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 18:45:29.290171 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"mutex_wait_us":212,"thread_start_us":323,"threads_started":1}
I20260620 18:45:29.290814 21801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d8494f33d7e46b6b106336f5a9242d1" candidate_uuid: "6f8524c2278f4cd29d9f88ba41228cc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d8f23504b8141be866b091a88bd6ace" is_pre_election: true
I20260620 18:45:29.291417 21801 raft_consensus.cc:2454] T 5d8494f33d7e46b6b106336f5a9242d1 P 7d8f23504b8141be866b091a88bd6ace [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f8524c2278f4cd29d9f88ba41228cc6 in term 0.
I20260620 18:45:29.291622 21667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d8494f33d7e46b6b106336f5a9242d1" candidate_uuid: "6f8524c2278f4cd29d9f88ba41228cc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1f9f638263945d7b85ff6c2c237c2f1" is_pre_election: true
I20260620 18:45:29.292163 21667 raft_consensus.cc:2454] T 5d8494f33d7e46b6b106336f5a9242d1 P e1f9f638263945d7b85ff6c2c237c2f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f8524c2278f4cd29d9f88ba41228cc6 in term 0.
I20260620 18:45:29.292120 21870 leader_election.cc:302] T 5d8494f33d7e46b6b106336f5a9242d1 P 6f8524c2278f4cd29d9f88ba41228cc6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f8524c2278f4cd29d9f88ba41228cc6, 7d8f23504b8141be866b091a88bd6ace; no voters:
I20260620 18:45:29.292657 22005 raft_consensus.cc:2781] T 5d8494f33d7e46b6b106336f5a9242d1 P 6f8524c2278f4cd29d9f88ba41228cc6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:45:29.292810 22005 raft_consensus.cc:482] T 5d8494f33d7e46b6b106336f5a9242d1 P 6f8524c2278f4cd29d9f88ba41228cc6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:45:29.292919 22005 raft_consensus.cc:3037] T 5d8494f33d7e46b6b106336f5a9242d1 P 6f8524c2278f4cd29d9f88ba41228cc6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:45:29.295346 22005 raft_consensus.cc:504] T 5d8494f33d7e46b6b106336f5a9242d1 P 6f8524c2278f4cd29d9f88ba41228cc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d8f23504b8141be866b091a88bd6ace" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 34311 } } peers { permanent_uuid: "e1f9f638263945d7b85ff6c2c237c2f1" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 41609 } } peers { permanent_uuid: "6f8524c2278f4cd29d9f88ba41228cc6" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 43087 } }
I20260620 18:45:29.295930 22005 leader_election.cc:288] T 5d8494f33d7e46b6b106336f5a9242d1 P 6f8524c2278f4cd29d9f88ba41228cc6 [CANDIDATE]: Term 1 election: Requested vote from peers 7d8f23504b8141be866b091a88bd6ace (127.5.225.130:34311), e1f9f638263945d7b85ff6c2c237c2f1 (127.5.225.129:41609)
I20260620 18:45:29.296319 21801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d8494f33d7e46b6b106336f5a9242d1" candidate_uuid: "6f8524c2278f4cd29d9f88ba41228cc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d8f23504b8141be866b091a88bd6ace"
I20260620 18:45:29.296414 21667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d8494f33d7e46b6b106336f5a9242d1" candidate_uuid: "6f8524c2278f4cd29d9f88ba41228cc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1f9f638263945d7b85ff6c2c237c2f1"
I20260620 18:45:29.296535 21801 raft_consensus.cc:3037] T 5d8494f33d7e46b6b106336f5a9242d1 P 7d8f23504b8141be866b091a88bd6ace [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:45:29.296658 21667 raft_consensus.cc:3037] T 5d8494f33d7e46b6b106336f5a9242d1 P e1f9f638263945d7b85ff6c2c237c2f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:45:29.298905 21801 raft_consensus.cc:2454] T 5d8494f33d7e46b6b106336f5a9242d1 P 7d8f23504b8141be866b091a88bd6ace [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f8524c2278f4cd29d9f88ba41228cc6 in term 1.
I20260620 18:45:29.299026 21667 raft_consensus.cc:2454] T 5d8494f33d7e46b6b106336f5a9242d1 P e1f9f638263945d7b85ff6c2c237c2f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f8524c2278f4cd29d9f88ba41228cc6 in term 1.
I20260620 18:45:29.299295 21870 leader_election.cc:302] T 5d8494f33d7e46b6b106336f5a9242d1 P 6f8524c2278f4cd29d9f88ba41228cc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f8524c2278f4cd29d9f88ba41228cc6, 7d8f23504b8141be866b091a88bd6ace; no voters:
I20260620 18:45:29.299578 22005 raft_consensus.cc:2781] T 5d8494f33d7e46b6b106336f5a9242d1 P 6f8524c2278f4cd29d9f88ba41228cc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:45:29.300036 22005 raft_consensus.cc:686] T 5d8494f33d7e46b6b106336f5a9242d1 P 6f8524c2278f4cd29d9f88ba41228cc6 [term 1 LEADER]: Becoming Leader. State: Replica: 6f8524c2278f4cd29d9f88ba41228cc6, State: Running, Role: LEADER
I20260620 18:45:29.300585 22005 consensus_queue.cc:237] T 5d8494f33d7e46b6b106336f5a9242d1 P 6f8524c2278f4cd29d9f88ba41228cc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d8f23504b8141be866b091a88bd6ace" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 34311 } } peers { permanent_uuid: "e1f9f638263945d7b85ff6c2c237c2f1" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 41609 } } peers { permanent_uuid: "6f8524c2278f4cd29d9f88ba41228cc6" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 43087 } }
W20260620 18:45:29.303980 21713 tablet.cc:2369] T 5d8494f33d7e46b6b106336f5a9242d1 P e1f9f638263945d7b85ff6c2c237c2f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:45:29.304984 21522 catalog_manager.cc:5495] T 5d8494f33d7e46b6b106336f5a9242d1 P 6f8524c2278f4cd29d9f88ba41228cc6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f8524c2278f4cd29d9f88ba41228cc6 (127.5.225.131). New cstate: current_term: 1 leader_uuid: "6f8524c2278f4cd29d9f88ba41228cc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d8f23504b8141be866b091a88bd6ace" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 34311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1f9f638263945d7b85ff6c2c237c2f1" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 41609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f8524c2278f4cd29d9f88ba41228cc6" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 43087 } health_report { overall_health: HEALTHY } } }
W20260620 18:45:29.312177 21847 tablet.cc:2369] T 5d8494f33d7e46b6b106336f5a9242d1 P 7d8f23504b8141be866b091a88bd6ace: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:45:29.318732 21980 tablet.cc:2369] T 5d8494f33d7e46b6b106336f5a9242d1 P 6f8524c2278f4cd29d9f88ba41228cc6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:45:29.346431 21996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:29.340853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51284 (local address 127.5.225.131:43087)
0620 18:45:29.341085 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:29.341091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:29.341110 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:45:29.341361 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:29.341366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:29.341445 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:29.341600 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:45:29.342003 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:29.342811 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:29.344293 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:29.344701 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:29.344841 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:29.345340 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:45:29.345848 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:45:29.346016 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:29.346064 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 18:45:29.346140 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 18:45:29.354414 21667 raft_consensus.cc:1261] T 5d8494f33d7e46b6b106336f5a9242d1 P e1f9f638263945d7b85ff6c2c237c2f1 [term 1 FOLLOWER]: Refusing update from remote peer 6f8524c2278f4cd29d9f88ba41228cc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:45:29.354411 21801 raft_consensus.cc:1261] T 5d8494f33d7e46b6b106336f5a9242d1 P 7d8f23504b8141be866b091a88bd6ace [term 1 FOLLOWER]: Refusing update from remote peer 6f8524c2278f4cd29d9f88ba41228cc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:45:29.355566 22007 consensus_queue.cc:1035] T 5d8494f33d7e46b6b106336f5a9242d1 P 6f8524c2278f4cd29d9f88ba41228cc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1f9f638263945d7b85ff6c2c237c2f1" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 41609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:45:29.355958 22005 consensus_queue.cc:1035] T 5d8494f33d7e46b6b106336f5a9242d1 P 6f8524c2278f4cd29d9f88ba41228cc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d8f23504b8141be866b091a88bd6ace" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 34311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:45:29.366747 22010 mvcc.cc:205] Tried to move back new op lower bound from 7298994705825587200 to 7298994705619513344. Current Snapshot: MvccSnapshot[applied={T|T < 7298994705825587200}]
I20260620 18:45:29.369455 22011 mvcc.cc:205] Tried to move back new op lower bound from 7298994705825587200 to 7298994705619513344. Current Snapshot: MvccSnapshot[applied={T|T < 7298994705825587200}]
I20260620 18:45:29.382107 21523 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:45:29.382439 21523 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:45:29.386046 21523 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049: Sending DeleteTablet for 3 replicas of tablet 5d8494f33d7e46b6b106336f5a9242d1
I20260620 18:45:29.387215 21647 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d8494f33d7e46b6b106336f5a9242d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:45:29 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:33684
I20260620 18:45:29.387259 21781 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d8494f33d7e46b6b106336f5a9242d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:45:29 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:39456
I20260620 18:45:29.388120 22021 tablet_replica.cc:330] T 5d8494f33d7e46b6b106336f5a9242d1 P 7d8f23504b8141be866b091a88bd6ace: stopping tablet replica
I20260620 18:45:29.388018 21914 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d8494f33d7e46b6b106336f5a9242d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:45:29 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:51268
I20260620 18:45:29.388526 22021 raft_consensus.cc:2229] T 5d8494f33d7e46b6b106336f5a9242d1 P 7d8f23504b8141be866b091a88bd6ace [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:45:29.388993 22021 raft_consensus.cc:2258] T 5d8494f33d7e46b6b106336f5a9242d1 P 7d8f23504b8141be866b091a88bd6ace [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:45:29.389314 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 21584
I20260620 18:45:29.391340 22021 ts_tablet_manager.cc:1905] T 5d8494f33d7e46b6b106336f5a9242d1 P 7d8f23504b8141be866b091a88bd6ace: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:45:29.399317 22020 tablet_replica.cc:330] T 5d8494f33d7e46b6b106336f5a9242d1 P e1f9f638263945d7b85ff6c2c237c2f1: stopping tablet replica
I20260620 18:45:29.399755 22020 raft_consensus.cc:2229] T 5d8494f33d7e46b6b106336f5a9242d1 P e1f9f638263945d7b85ff6c2c237c2f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:45:29.400256 22020 raft_consensus.cc:2258] T 5d8494f33d7e46b6b106336f5a9242d1 P e1f9f638263945d7b85ff6c2c237c2f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:45:29.405282 22021 ts_tablet_manager.cc:1918] T 5d8494f33d7e46b6b106336f5a9242d1 P 7d8f23504b8141be866b091a88bd6ace: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:45:29.405452 22021 log.cc:1196] T 5d8494f33d7e46b6b106336f5a9242d1 P 7d8f23504b8141be866b091a88bd6ace: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/wal/wals/5d8494f33d7e46b6b106336f5a9242d1
I20260620 18:45:29.407498 22020 ts_tablet_manager.cc:1905] T 5d8494f33d7e46b6b106336f5a9242d1 P e1f9f638263945d7b85ff6c2c237c2f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:45:29.412989 22022 tablet_replica.cc:330] T 5d8494f33d7e46b6b106336f5a9242d1 P 6f8524c2278f4cd29d9f88ba41228cc6: stopping tablet replica
I20260620 18:45:29.413344 22021 ts_tablet_manager.cc:1939] T 5d8494f33d7e46b6b106336f5a9242d1 P 7d8f23504b8141be866b091a88bd6ace: Deleting consensus metadata
I20260620 18:45:29.413482 22022 raft_consensus.cc:2229] T 5d8494f33d7e46b6b106336f5a9242d1 P 6f8524c2278f4cd29d9f88ba41228cc6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:45:29.414273 22022 raft_consensus.cc:2258] T 5d8494f33d7e46b6b106336f5a9242d1 P 6f8524c2278f4cd29d9f88ba41228cc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:45:29.415518 21510 catalog_manager.cc:4841] TS 7d8f23504b8141be866b091a88bd6ace (127.5.225.130:34311): tablet 5d8494f33d7e46b6b106336f5a9242d1 (table test-table [id=457ed6eaa9614e1f8758d5897ea01f21]) successfully deleted
I20260620 18:45:29.416554 22022 ts_tablet_manager.cc:1905] T 5d8494f33d7e46b6b106336f5a9242d1 P 6f8524c2278f4cd29d9f88ba41228cc6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:45:29.417740 22020 ts_tablet_manager.cc:1918] T 5d8494f33d7e46b6b106336f5a9242d1 P e1f9f638263945d7b85ff6c2c237c2f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:45:29.417899 22020 log.cc:1196] T 5d8494f33d7e46b6b106336f5a9242d1 P e1f9f638263945d7b85ff6c2c237c2f1: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/wal/wals/5d8494f33d7e46b6b106336f5a9242d1
I20260620 18:45:29.418462 22020 ts_tablet_manager.cc:1939] T 5d8494f33d7e46b6b106336f5a9242d1 P e1f9f638263945d7b85ff6c2c237c2f1: Deleting consensus metadata
I20260620 18:45:29.420235 21509 catalog_manager.cc:4841] TS e1f9f638263945d7b85ff6c2c237c2f1 (127.5.225.129:41609): tablet 5d8494f33d7e46b6b106336f5a9242d1 (table test-table [id=457ed6eaa9614e1f8758d5897ea01f21]) successfully deleted
I20260620 18:45:29.424089 22022 ts_tablet_manager.cc:1918] T 5d8494f33d7e46b6b106336f5a9242d1 P 6f8524c2278f4cd29d9f88ba41228cc6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Jun 20 18:45:29 dist-test-slave-k6lg krb5kdc[21478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.129@KRBTEST.COM
I20260620 18:45:29.424256 22022 log.cc:1196] T 5d8494f33d7e46b6b106336f5a9242d1 P 6f8524c2278f4cd29d9f88ba41228cc6: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/wal/wals/5d8494f33d7e46b6b106336f5a9242d1
I20260620 18:45:29.424813 22022 ts_tablet_manager.cc:1939] T 5d8494f33d7e46b6b106336f5a9242d1 P 6f8524c2278f4cd29d9f88ba41228cc6: Deleting consensus metadata
I20260620 18:45:29.426549 21510 catalog_manager.cc:4841] TS 6f8524c2278f4cd29d9f88ba41228cc6 (127.5.225.131:43087): tablet 5d8494f33d7e46b6b106336f5a9242d1 (table test-table [id=457ed6eaa9614e1f8758d5897ea01f21]) successfully deleted
I20260620 18:45:29.431603 21994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:29.390862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33696 (local address 127.5.225.129:41609)
0620 18:45:29.391098 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:29.391107 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:29.412103 (+ 20996us) server_negotiation.cc:407] Connection header received
0620 18:45:29.412344 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:29.412352 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:29.412458 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:29.412571 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:29.415376 (+ 2805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:29.416482 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:29.418953 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:29.419626 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:29.419723 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:29.425623 (+ 5900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:29.425638 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:29.425656 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:29.425704 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:29.428700 (+ 2996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:29.429556 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:29.429563 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:29.429565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:29.429689 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:29.430119 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:29.430122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:29.430124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:29.430520 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:29.430788 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:29.431139 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 18:45:29.431296 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:45:29.432411 21707 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:29.580642 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 21584
I20260620 18:45:29.601476 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 21718
Jun 20 18:45:29 dist-test-slave-k6lg krb5kdc[21478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.130@KRBTEST.COM
I20260620 18:45:29.620466 21992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:29.602427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39466 (local address 127.5.225.130:34311)
0620 18:45:29.602618 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:29.602622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:29.602640 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:45:29.602781 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:29.602785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:29.602856 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:29.602939 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:29.604628 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:29.605513 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:29.607616 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:29.608017 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:29.608115 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:29.613539 (+ 5424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:29.613559 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:29.613580 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:29.613625 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:29.617445 (+ 3820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:29.618203 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:29.618213 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:29.618217 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:29.618365 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:29.618781 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:29.618787 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:29.618790 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:29.619291 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:29.619589 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:29.619865 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 18:45:29.620082 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:45:29.621318 21841 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:29.713294 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 21718
I20260620 18:45:29.731729 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 21851
Jun 20 18:45:29 dist-test-slave-k6lg krb5kdc[21478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.131@KRBTEST.COM
I20260620 18:45:29.755676 21996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:29.732616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51290 (local address 127.5.225.131:43087)
0620 18:45:29.732923 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:29.732932 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:29.735311 (+ 2379us) server_negotiation.cc:407] Connection header received
0620 18:45:29.735381 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:29.735385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:29.735463 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:29.735619 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:29.737682 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:29.739004 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:29.741261 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:29.742110 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:29.743147 (+ 1037us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:29.749598 (+ 6451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:29.749611 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:29.749626 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:29.749672 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:29.752674 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:29.753373 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:29.753382 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:29.753396 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:29.753574 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:29.753957 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:29.753962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:29.753966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:29.754395 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:29.754752 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:29.755059 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 18:45:29.755287 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260620 18:45:29.756398 21974 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:29.881706 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 21851
I20260620 18:45:29.903014 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 21491
I20260620 18:45:29.904160 21553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:30.018657 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 21491
I20260620 18:45:50.041376 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:45289
--webserver_interface=127.5.225.190
--webserver_port=45995
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:35541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:45289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:45:50.371456 22028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:50.371799 22028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:50.371970 22028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:50.379204 22028 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 18:45:50.379357 22028 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 18:45:50.379453 22028 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 18:45:50.379558 22028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:45:50.379657 22028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:50.379761 22028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:50.379850 22028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:45:50.379935 22028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:45:50.389804 22028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:35541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:45289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:45289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=45995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:45:50.391209 22028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:50.392539 22028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:50.415504 22034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:50.415483 22037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:50.415584 22035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:50.416577 22028 server_base.cc:1034] running on GCE node
I20260620 18:45:50.417438 22028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:50.419049 22028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:50.420342 22028 hybrid_clock.cc:648] HybridClock initialized: now 1781981150420256 us; error 38 us; skew 500 ppm
I20260620 18:45:50.420766 22028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:45:50 dist-test-slave-k6lg krb5kdc[21478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981150, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:45:50.430122 22028 init.cc:377] Logged in from keytab as kudu/127.5.225.190@KRBTEST.COM (short username kudu)
I20260620 18:45:50.431861 22028 webserver.cc:460] Webserver started at http://127.5.225.190:45995/ using document root <none> and password file <none>
I20260620 18:45:50.432622 22028 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:50.432724 22028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:50.438853 22028 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 18:45:50.441764 22044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:50.443218 22028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:45:50.443368 22028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "0f9a644e0e0146b4a66ef256b1f7a049"
format_stamp: "Formatted at 2026-06-20 18:45:27 on dist-test-slave-k6lg"
server_key: "25f877fe481f72bdacc5bfa67cb852ac"
server_key_iv: "ca562b88a2576b4771e3a3fba1827c71"
server_key_version: "encryptionkey@0"
I20260620 18:45:50.444380 22028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:50.471830 22028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:50.476404 22028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:50.476646 22028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:50.495594 22028 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:45289
I20260620 18:45:50.495630 22095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:45289 every 8 connection(s)
I20260620 18:45:50.497083 22028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:45:50.504056 22096 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:45:50.505057 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 22028
I20260620 18:45:50.506131 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.129:41609
--local_ip_for_outbound_sockets=127.5.225.129
--tserver_master_addrs=127.5.225.190:45289
--webserver_port=34483
--webserver_interface=127.5.225.129
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:35541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:45:50.509050 22096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049: Bootstrap starting.
I20260620 18:45:50.530997 22096 log.cc:824] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049: Log is configured to *not* fsync() on all Append() calls
I20260620 18:45:50.543609 22096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:45:50.544229 22096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049: Bootstrap complete.
I20260620 18:45:50.554286 22096 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f9a644e0e0146b4a66ef256b1f7a049" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 45289 } }
I20260620 18:45:50.555497 22096 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f9a644e0e0146b4a66ef256b1f7a049, State: Initialized, Role: FOLLOWER
I20260620 18:45:50.556102 22096 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f9a644e0e0146b4a66ef256b1f7a049" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 45289 } }
I20260620 18:45:50.556341 22096 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:45:50.556476 22096 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:45:50.556665 22096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:45:50.560513 22096 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f9a644e0e0146b4a66ef256b1f7a049" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 45289 } }
I20260620 18:45:50.561038 22096 leader_election.cc:302] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f9a644e0e0146b4a66ef256b1f7a049; no voters:
I20260620 18:45:50.561700 22096 leader_election.cc:288] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 18:45:50.561841 22100 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 18:45:50.563118 22100 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [term 2 LEADER]: Becoming Leader. State: Replica: 0f9a644e0e0146b4a66ef256b1f7a049, State: Running, Role: LEADER
I20260620 18:45:50.563664 22100 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f9a644e0e0146b4a66ef256b1f7a049" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 45289 } }
I20260620 18:45:50.564219 22096 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:45:50.566349 22101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0f9a644e0e0146b4a66ef256b1f7a049" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f9a644e0e0146b4a66ef256b1f7a049" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 45289 } } }
I20260620 18:45:50.566757 22101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [sys.catalog]: This master's current role is: LEADER
I20260620 18:45:50.568365 22105 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:45:50.568624 22102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f9a644e0e0146b4a66ef256b1f7a049. Latest consensus state: current_term: 2 leader_uuid: "0f9a644e0e0146b4a66ef256b1f7a049" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f9a644e0e0146b4a66ef256b1f7a049" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 45289 } } }
I20260620 18:45:50.569016 22102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [sys.catalog]: This master's current role is: LEADER
I20260620 18:45:50.580691 22105 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:45:50.582235 22105 catalog_manager.cc:1259] Loaded cluster ID: ad104ace8f224e259c912cbb453d1f6b
I20260620 18:45:50.582417 22105 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:45:50.586519 22105 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:45:50.588102 22105 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049: Loaded TSK: 0
I20260620 18:45:50.623731 22105 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049: Generated new TSK 1
I20260620 18:45:50.624136 22105 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 18:45:50.859723 22098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:50.860106 22098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:50.860278 22098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:50.867817 22098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:50.867991 22098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:50.868166 22098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.129
I20260620 18:45:50.878098 22098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:35541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.129:41609
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.225.129
--webserver_port=34483
--tserver_master_addrs=127.5.225.190:45289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.129
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:45:50.879454 22098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:50.880791 22098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:50.898653 22123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:50.899278 22124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:50.899637 22126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:50.900897 22098 server_base.cc:1034] running on GCE node
I20260620 18:45:50.901594 22098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:50.902688 22098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:50.903928 22098 hybrid_clock.cc:648] HybridClock initialized: now 1781981150903880 us; error 42 us; skew 500 ppm
I20260620 18:45:50.904269 22098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:45:50 dist-test-slave-k6lg krb5kdc[21478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981150, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:45:50.907514 22098 init.cc:377] Logged in from keytab as kudu/127.5.225.129@KRBTEST.COM (short username kudu)
I20260620 18:45:50.909060 22098 webserver.cc:460] Webserver started at http://127.5.225.129:34483/ using document root <none> and password file <none>
I20260620 18:45:50.909775 22098 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:50.909942 22098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:50.915261 22098 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:45:50.917925 22133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:50.919325 22098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:45:50.919476 22098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/wal
uuid: "e1f9f638263945d7b85ff6c2c237c2f1"
format_stamp: "Formatted at 2026-06-20 18:45:27 on dist-test-slave-k6lg"
server_key: "f0a4c2e6ed04336be52dda6992a19228"
server_key_iv: "ec7e732668ae09a39358427b02d70eb5"
server_key_version: "encryptionkey@0"
I20260620 18:45:50.920387 22098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:50.935686 22098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:50.940007 22098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:50.940276 22098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:50.941246 22098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:45:50.942616 22098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:45:50.942703 22098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:50.942778 22098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:45:50.942832 22098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:50.975746 22098 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.129:41609
I20260620 18:45:50.975800 22245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.129:41609 every 8 connection(s)
I20260620 18:45:50.977140 22098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
I20260620 18:45:50.982692 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 22098
I20260620 18:45:50.983461 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.130:34311
--local_ip_for_outbound_sockets=127.5.225.130
--tserver_master_addrs=127.5.225.190:45289
--webserver_port=43493
--webserver_interface=127.5.225.130
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:35541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:45:50 dist-test-slave-k6lg krb5kdc[21478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981150, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:45:51.005342 22249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:50.980297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.129:41993)
0620 18:45:50.981202 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:50.981254 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:50.983593 (+ 2339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:50.985124 (+ 1531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:50.985137 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:50.985939 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:50.986717 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:50.986742 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:50.988543 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:50.988551 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:50.990773 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:50.990786 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:50.992689 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:50.992698 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:50.993009 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:50.993746 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:50.993779 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:50.996389 (+ 2610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:45:51.001083 (+ 4694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:51.001090 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:51.001102 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:51.001532 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:51.002037 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:51.002040 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:51.002042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:51.002299 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:51.003259 (+ 960us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:45:51.003265 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:45:51.003665 (+ 400us) client_negotiation.cc:769] Sending connection context
0620 18:45:51.003906 (+ 241us) client_negotiation.cc:240] Negotiation successful
0620 18:45:51.004294 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":349,"threads_started":1}
I20260620 18:45:51.006237 22248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:50.980098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:41993 (local address 127.5.225.190:45289)
0620 18:45:50.980630 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:50.980642 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:50.981435 (+ 793us) server_negotiation.cc:407] Connection header received
0620 18:45:50.984190 (+ 2755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:50.984205 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:50.984594 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:50.984911 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:50.986934 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:50.988364 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:50.990921 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:50.992114 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:50.992314 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:50.996714 (+ 4400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:50.996728 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:50.996751 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:50.996807 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:51.000917 (+ 4110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:51.001694 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:51.001705 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:51.001716 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:51.001926 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:51.002430 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:51.002438 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:51.002444 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:51.003115 (+ 671us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:51.003353 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:51.004214 (+ 861us) server_negotiation.cc:299] Negotiation successful
0620 18:45:51.004724 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":235,"threads_started":1}
I20260620 18:45:51.009255 22246 heartbeater.cc:347] Connected to a master server at 127.5.225.190:45289
I20260620 18:45:51.009611 22246 heartbeater.cc:464] Registering TS with master...
I20260620 18:45:51.010634 22246 heartbeater.cc:511] Master 127.5.225.190:45289 requested a full tablet report, sending...
I20260620 18:45:51.013069 22061 ts_manager.cc:194] Registered new tserver with Master: e1f9f638263945d7b85ff6c2c237c2f1 (127.5.225.129:41609)
I20260620 18:45:51.014674 22061 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.129@KRBTEST.COM'} at 127.5.225.129:41993
W20260620 18:45:51.320219 22250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:51.320487 22250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:51.320639 22250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:51.327983 22250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:51.328117 22250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:51.328313 22250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.130
I20260620 18:45:51.337914 22250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:35541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.130:34311
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.225.130
--webserver_port=43493
--tserver_master_addrs=127.5.225.190:45289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.130
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:45:51.339198 22250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:51.340473 22250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:51.353731 22255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:51.353709 22256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:51.353912 22258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:51.354519 22250 server_base.cc:1034] running on GCE node
I20260620 18:45:51.355053 22250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:51.355870 22250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:51.357096 22250 hybrid_clock.cc:648] HybridClock initialized: now 1781981151357040 us; error 49 us; skew 500 ppm
I20260620 18:45:51.357410 22250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:45:51 dist-test-slave-k6lg krb5kdc[21478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981151, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:45:51.360478 22250 init.cc:377] Logged in from keytab as kudu/127.5.225.130@KRBTEST.COM (short username kudu)
I20260620 18:45:51.361894 22250 webserver.cc:460] Webserver started at http://127.5.225.130:43493/ using document root <none> and password file <none>
I20260620 18:45:51.362612 22250 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:51.362771 22250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:51.367942 22250 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 18:45:51.371024 22265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:51.372426 22250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:45:51.372607 22250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/wal
uuid: "7d8f23504b8141be866b091a88bd6ace"
format_stamp: "Formatted at 2026-06-20 18:45:28 on dist-test-slave-k6lg"
server_key: "1431d5002b2e2fba1bb57522ac2fef17"
server_key_iv: "e6d97a8d9a471bb66ed86867698a539d"
server_key_version: "encryptionkey@0"
I20260620 18:45:51.373595 22250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:51.384478 22250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:51.388301 22250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:51.388577 22250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:51.389523 22250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:45:51.390951 22250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:45:51.391081 22250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:51.391219 22250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:45:51.391309 22250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:51.428671 22250 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.130:34311
I20260620 18:45:51.428758 22377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.130:34311 every 8 connection(s)
I20260620 18:45:51.430335 22250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
I20260620 18:45:51.431149 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 22250
I20260620 18:45:51.431917 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.131:43087
--local_ip_for_outbound_sockets=127.5.225.131
--tserver_master_addrs=127.5.225.190:45289
--webserver_port=36111
--webserver_interface=127.5.225.131
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:35541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:45:51 dist-test-slave-k6lg krb5kdc[21478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981151, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:45:51.453015 22248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:51.433146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:51495 (local address 127.5.225.190:45289)
0620 18:45:51.433330 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:51.433335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:51.434898 (+ 1563us) server_negotiation.cc:407] Connection header received
0620 18:45:51.436873 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:51.436880 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:51.436981 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:51.437095 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:51.438709 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:51.439830 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:51.441381 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:51.442206 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:51.442345 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:51.445884 (+ 3539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:51.445894 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:51.445902 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:51.445949 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:51.449654 (+ 3705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:51.450441 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:51.450445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:51.450448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:51.450582 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:51.451114 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:51.451117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:51.451119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:51.451393 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:51.451521 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:51.452479 (+ 958us) server_negotiation.cc:299] Negotiation successful
0620 18:45:51.452717 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 18:45:51.454667 22380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:51.433741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.130:51495)
0620 18:45:51.434724 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:51.434766 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:51.436592 (+ 1826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:51.437331 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:51.437341 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:51.437898 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:51.438531 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:51.438549 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:51.439998 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:51.440003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:51.441224 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:51.441236 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:51.442418 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:51.442423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:51.442622 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:51.443328 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:51.443350 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:51.445649 (+ 2299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:45:51.449783 (+ 4134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:51.449789 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:51.449802 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:51.450308 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:51.450700 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:51.450708 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:51.450716 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:51.450992 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:51.451531 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:45:51.451542 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:45:51.452049 (+ 507us) client_negotiation.cc:769] Sending connection context
0620 18:45:51.452407 (+ 358us) client_negotiation.cc:240] Negotiation successful
0620 18:45:51.453064 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":477,"threads_started":1}
I20260620 18:45:51.456157 22378 heartbeater.cc:347] Connected to a master server at 127.5.225.190:45289
I20260620 18:45:51.456511 22378 heartbeater.cc:464] Registering TS with master...
I20260620 18:45:51.457437 22378 heartbeater.cc:511] Master 127.5.225.190:45289 requested a full tablet report, sending...
I20260620 18:45:51.459167 22061 ts_manager.cc:194] Registered new tserver with Master: 7d8f23504b8141be866b091a88bd6ace (127.5.225.130:34311)
I20260620 18:45:51.460007 22061 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.130@KRBTEST.COM'} at 127.5.225.130:51495
W20260620 18:45:51.762916 22381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:51.763190 22381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:51.763388 22381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:51.770829 22381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:51.770959 22381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:51.771153 22381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.131
I20260620 18:45:51.780779 22381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:35541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.131:43087
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.225.131
--webserver_port=36111
--tserver_master_addrs=127.5.225.190:45289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.131
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:45:51.782050 22381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:51.783353 22381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:51.795987 22387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:51.796041 22386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:51.796375 22389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:51.796821 22381 server_base.cc:1034] running on GCE node
I20260620 18:45:51.797390 22381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:51.798372 22381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:51.799621 22381 hybrid_clock.cc:648] HybridClock initialized: now 1781981151799581 us; error 48 us; skew 500 ppm
I20260620 18:45:51.800011 22381 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:45:51 dist-test-slave-k6lg krb5kdc[21478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981151, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:45:51.803272 22381 init.cc:377] Logged in from keytab as kudu/127.5.225.131@KRBTEST.COM (short username kudu)
I20260620 18:45:51.804728 22381 webserver.cc:460] Webserver started at http://127.5.225.131:36111/ using document root <none> and password file <none>
I20260620 18:45:51.805397 22381 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:51.805557 22381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:51.811012 22381 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:45:51.813743 22396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:51.815209 22381 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:45:51.815392 22381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/wal
uuid: "6f8524c2278f4cd29d9f88ba41228cc6"
format_stamp: "Formatted at 2026-06-20 18:45:28 on dist-test-slave-k6lg"
server_key: "ef8f8c8c7e070ddf046e4f98e6e8321c"
server_key_iv: "9f0c21d2af9a17cd04dda8d1b6efe893"
server_key_version: "encryptionkey@0"
I20260620 18:45:51.816371 22381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:51.837497 22381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:51.841955 22381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:51.842304 22381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:51.843329 22381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:45:51.844671 22381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:45:51.844805 22381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:51.844946 22381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:45:51.845036 22381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:51.879945 22381 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.131:43087
I20260620 18:45:51.880030 22508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.131:43087 every 8 connection(s)
I20260620 18:45:51.881460 22381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
I20260620 18:45:51.888258 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 22381
Jun 20 18:45:51 dist-test-slave-k6lg krb5kdc[21478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981151, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:45:51.902209 22248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:51.884080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:42255 (local address 127.5.225.190:45289)
0620 18:45:51.884257 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:51.884262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:51.885330 (+ 1068us) server_negotiation.cc:407] Connection header received
0620 18:45:51.887284 (+ 1954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:51.887290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:51.887380 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:51.887491 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:51.889445 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:51.890372 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:51.891897 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:51.892323 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:51.892454 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:51.896180 (+ 3726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:51.896184 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:51.896188 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:51.896218 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:51.899203 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:51.899910 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:51.899916 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:51.899920 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:51.900058 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:51.900483 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:51.900489 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:51.900493 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:51.900781 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:51.900935 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:51.901597 (+ 662us) server_negotiation.cc:299] Negotiation successful
0620 18:45:51.901805 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:45:51.903162 22511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:51.884476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.131:42255)
0620 18:45:51.885162 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:51.885206 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:51.887011 (+ 1805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:51.887733 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:51.887741 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:51.888548 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:51.889280 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:51.889299 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:51.890545 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:51.890554 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:51.891746 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:51.891757 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:51.892570 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:51.892578 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:51.892802 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:51.893451 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:51.893477 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:51.895977 (+ 2500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:45:51.899338 (+ 3361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:51.899347 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:51.899363 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:51.899784 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:51.900188 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:51.900193 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:51.900197 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:51.900375 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:51.900907 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:45:51.900915 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:45:51.901252 (+ 337us) client_negotiation.cc:769] Sending connection context
0620 18:45:51.901512 (+ 260us) client_negotiation.cc:240] Negotiation successful
0620 18:45:51.902028 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":264,"threads_started":1}
I20260620 18:45:51.904738 22509 heartbeater.cc:347] Connected to a master server at 127.5.225.190:45289
I20260620 18:45:51.905071 22509 heartbeater.cc:464] Registering TS with master...
I20260620 18:45:51.905809 22509 heartbeater.cc:511] Master 127.5.225.190:45289 requested a full tablet report, sending...
I20260620 18:45:51.907361 22061 ts_manager.cc:194] Registered new tserver with Master: 6f8524c2278f4cd29d9f88ba41228cc6 (127.5.225.131:43087)
I20260620 18:45:51.908526 22061 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.131@KRBTEST.COM'} at 127.5.225.131:42255
I20260620 18:45:51.908898 22513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:51.889319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58088 (local address 127.5.225.190:45289)
0620 18:45:51.895168 (+ 5849us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:51.895172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:51.895185 (+ 13us) server_negotiation.cc:407] Connection header received
0620 18:45:51.895220 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:51.895224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:51.895283 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:51.895408 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:51.897102 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:51.898327 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:51.899806 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:51.900259 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:51.900393 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:51.903292 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:51.903298 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:51.903302 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:51.903335 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:51.906115 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:51.906858 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:51.906864 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:51.906867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:51.907024 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:51.907396 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:51.907400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:51.907403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:51.907775 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:51.908049 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:51.908276 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 18:45:51.908480 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5759,"spinlock_wait_cycles":16256,"thread_start_us":5627,"threads_started":1}
I20260620 18:45:51.911422 6022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:45:51.917382 22513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:51.912396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58092 (local address 127.5.225.190:45289)
0620 18:45:51.912631 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:51.912637 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:51.912813 (+ 176us) server_negotiation.cc:407] Connection header received
0620 18:45:51.912957 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:51.912961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:51.913089 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:51.913199 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:45:51.913653 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:51.914392 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:51.915805 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:51.916197 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:51.916283 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:51.916743 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:45:51.916929 (+ 186us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 18:45:51.917175 (+ 246us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58092: token expired
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 18:45:51.917500 22513 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58092: token expired
W20260620 18:45:51.918064 21984 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.5.225.190:45289)
I20260620 18:45:51.931099 22513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:51.918548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58106 (local address 127.5.225.190:45289)
0620 18:45:51.918706 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:51.918713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:51.918734 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:45:51.918904 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:51.918908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:51.918996 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:51.919097 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:51.920587 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:51.921271 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:51.922722 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:51.923187 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:51.923334 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:51.925968 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:51.925974 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:51.925978 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:51.926018 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:51.928438 (+ 2420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:51.929158 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:51.929164 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:51.929168 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:51.929295 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:51.929656 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:51.929661 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:51.929665 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:51.929974 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:51.930259 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:51.930555 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 18:45:51.930773 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 18:45:51.934557 22061 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:45:51.941617 22061 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:45:51.963157 22520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:51.953490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59160 (local address 127.5.225.130:34311)
0620 18:45:51.954333 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:51.954343 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:51.954382 (+ 39us) server_negotiation.cc:407] Connection header received
0620 18:45:51.955115 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:51.955129 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:51.955441 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:51.955997 (+ 556us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:45:51.956456 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:51.958217 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:51.960337 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:51.961881 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:51.962024 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:51.962270 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:51.962642 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 18:45:51.962817 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":510,"threads_started":1}
I20260620 18:45:51.963229 22519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:51.953076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.130:34311 (local address 127.0.0.1:59160)
0620 18:45:51.953874 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:51.953924 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:51.954107 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:51.955697 (+ 1590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:51.955704 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:51.955735 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:45:51.956275 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:51.956294 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:51.958366 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:51.958372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:51.960182 (+ 1810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:51.960194 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:51.962134 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:51.962140 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:51.962392 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:51.962451 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 18:45:51.962593 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 18:45:51.962809 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":26,"thread_start_us":398,"threads_started":1}
I20260620 18:45:51.963579 22521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:51.954448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:41609 (local address 127.0.0.1:50290)
0620 18:45:51.955250 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:51.955269 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:51.955376 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:51.956473 (+ 1097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:51.956477 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:51.956508 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:45:51.956900 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:51.956912 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:51.958853 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:51.958858 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:51.960911 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:51.960928 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:51.962818 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:51.962824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:51.962987 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:51.963021 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 18:45:51.963089 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 18:45:51.963226 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":348,"threads_started":1}
I20260620 18:45:51.964612 22522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:51.954669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50290 (local address 127.5.225.129:41609)
0620 18:45:51.955877 (+ 1208us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:51.955888 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:51.955916 (+ 28us) server_negotiation.cc:407] Connection header received
0620 18:45:51.956007 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:51.956014 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:51.956258 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:51.957043 (+ 785us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:45:51.957241 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:51.958708 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:51.961105 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:51.962591 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:51.963889 (+ 1298us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:51.964059 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:51.964118 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 18:45:51.964275 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1041,"thread_start_us":927,"threads_started":1}
I20260620 18:45:51.965528 22524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:51.955080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.131:43087 (local address 127.0.0.1:34804)
0620 18:45:51.956776 (+ 1696us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:51.956793 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:51.956883 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:51.958052 (+ 1169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:51.958057 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:51.958078 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:45:51.958525 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:51.958543 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:51.960486 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:51.960492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:51.962598 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:51.962612 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:51.964718 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:51.964724 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:51.964891 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:51.964959 (+ 68us) client_negotiation.cc:769] Sending connection context
0620 18:45:51.965047 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 18:45:51.965174 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1593,"thread_start_us":1425,"threads_started":1}
I20260620 18:45:51.965765 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:51.954647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34804 (local address 127.5.225.131:43087)
0620 18:45:51.957278 (+ 2631us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:51.957286 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:51.957317 (+ 31us) server_negotiation.cc:407] Connection header received
0620 18:45:51.957406 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:51.957414 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:51.957636 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:51.957805 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:45:51.958704 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:51.960351 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:51.962775 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:51.964495 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:51.964632 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:51.964844 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:51.965124 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 18:45:51.965274 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2444,"thread_start_us":1670,"threads_started":1}
I20260620 18:45:51.967334 22313 tablet_service.cc:1467] Processing CreateTablet for tablet 0f6083e823d545349707e5cba99df387 (DEFAULT_TABLE table=test-table [id=83c562aa843c4ba3aaa024fec9222496]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:45:51.968803 22313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f6083e823d545349707e5cba99df387. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:45:51.969019 22181 tablet_service.cc:1467] Processing CreateTablet for tablet 0f6083e823d545349707e5cba99df387 (DEFAULT_TABLE table=test-table [id=83c562aa843c4ba3aaa024fec9222496]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:45:51.969218 22444 tablet_service.cc:1467] Processing CreateTablet for tablet 0f6083e823d545349707e5cba99df387 (DEFAULT_TABLE table=test-table [id=83c562aa843c4ba3aaa024fec9222496]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:45:51.970561 22444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f6083e823d545349707e5cba99df387. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:45:51.970623 22181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f6083e823d545349707e5cba99df387. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:45:51.988173 22527 tablet_bootstrap.cc:492] T 0f6083e823d545349707e5cba99df387 P 6f8524c2278f4cd29d9f88ba41228cc6: Bootstrap starting.
I20260620 18:45:51.990124 22525 tablet_bootstrap.cc:492] T 0f6083e823d545349707e5cba99df387 P 7d8f23504b8141be866b091a88bd6ace: Bootstrap starting.
I20260620 18:45:51.990897 22526 tablet_bootstrap.cc:492] T 0f6083e823d545349707e5cba99df387 P e1f9f638263945d7b85ff6c2c237c2f1: Bootstrap starting.
I20260620 18:45:51.993166 22527 tablet_bootstrap.cc:654] T 0f6083e823d545349707e5cba99df387 P 6f8524c2278f4cd29d9f88ba41228cc6: Neither blocks nor log segments found. Creating new log.
I20260620 18:45:51.993510 22525 tablet_bootstrap.cc:654] T 0f6083e823d545349707e5cba99df387 P 7d8f23504b8141be866b091a88bd6ace: Neither blocks nor log segments found. Creating new log.
I20260620 18:45:51.993839 22526 tablet_bootstrap.cc:654] T 0f6083e823d545349707e5cba99df387 P e1f9f638263945d7b85ff6c2c237c2f1: Neither blocks nor log segments found. Creating new log.
I20260620 18:45:51.994484 22527 log.cc:824] T 0f6083e823d545349707e5cba99df387 P 6f8524c2278f4cd29d9f88ba41228cc6: Log is configured to *not* fsync() on all Append() calls
I20260620 18:45:51.994665 22525 log.cc:824] T 0f6083e823d545349707e5cba99df387 P 7d8f23504b8141be866b091a88bd6ace: Log is configured to *not* fsync() on all Append() calls
I20260620 18:45:51.994918 22526 log.cc:824] T 0f6083e823d545349707e5cba99df387 P e1f9f638263945d7b85ff6c2c237c2f1: Log is configured to *not* fsync() on all Append() calls
I20260620 18:45:51.998679 22527 tablet_bootstrap.cc:492] T 0f6083e823d545349707e5cba99df387 P 6f8524c2278f4cd29d9f88ba41228cc6: No bootstrap required, opened a new log
I20260620 18:45:51.999046 22527 ts_tablet_manager.cc:1397] T 0f6083e823d545349707e5cba99df387 P 6f8524c2278f4cd29d9f88ba41228cc6: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260620 18:45:52.003947 22525 tablet_bootstrap.cc:492] T 0f6083e823d545349707e5cba99df387 P 7d8f23504b8141be866b091a88bd6ace: No bootstrap required, opened a new log
I20260620 18:45:52.004220 22525 ts_tablet_manager.cc:1397] T 0f6083e823d545349707e5cba99df387 P 7d8f23504b8141be866b091a88bd6ace: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260620 18:45:52.004273 22526 tablet_bootstrap.cc:492] T 0f6083e823d545349707e5cba99df387 P e1f9f638263945d7b85ff6c2c237c2f1: No bootstrap required, opened a new log
I20260620 18:45:52.004534 22526 ts_tablet_manager.cc:1397] T 0f6083e823d545349707e5cba99df387 P e1f9f638263945d7b85ff6c2c237c2f1: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260620 18:45:52.009729 22527 raft_consensus.cc:348] T 0f6083e823d545349707e5cba99df387 P 6f8524c2278f4cd29d9f88ba41228cc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d8f23504b8141be866b091a88bd6ace" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 34311 } } peers { permanent_uuid: "6f8524c2278f4cd29d9f88ba41228cc6" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 43087 } } peers { permanent_uuid: "e1f9f638263945d7b85ff6c2c237c2f1" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 41609 } }
I20260620 18:45:52.010226 22527 raft_consensus.cc:374] T 0f6083e823d545349707e5cba99df387 P 6f8524c2278f4cd29d9f88ba41228cc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:45:52.010329 22527 raft_consensus.cc:729] T 0f6083e823d545349707e5cba99df387 P 6f8524c2278f4cd29d9f88ba41228cc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f8524c2278f4cd29d9f88ba41228cc6, State: Initialized, Role: FOLLOWER
I20260620 18:45:52.010964 22527 consensus_queue.cc:260] T 0f6083e823d545349707e5cba99df387 P 6f8524c2278f4cd29d9f88ba41228cc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d8f23504b8141be866b091a88bd6ace" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 34311 } } peers { permanent_uuid: "6f8524c2278f4cd29d9f88ba41228cc6" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 43087 } } peers { permanent_uuid: "e1f9f638263945d7b85ff6c2c237c2f1" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 41609 } }
I20260620 18:45:52.012045 22509 heartbeater.cc:503] Master 127.5.225.190:45289 was elected leader, sending a full tablet report...
I20260620 18:45:52.012701 22526 raft_consensus.cc:348] T 0f6083e823d545349707e5cba99df387 P e1f9f638263945d7b85ff6c2c237c2f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d8f23504b8141be866b091a88bd6ace" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 34311 } } peers { permanent_uuid: "6f8524c2278f4cd29d9f88ba41228cc6" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 43087 } } peers { permanent_uuid: "e1f9f638263945d7b85ff6c2c237c2f1" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 41609 } }
I20260620 18:45:52.013010 22527 ts_tablet_manager.cc:1428] T 0f6083e823d545349707e5cba99df387 P 6f8524c2278f4cd29d9f88ba41228cc6: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260620 18:45:52.013098 22526 raft_consensus.cc:374] T 0f6083e823d545349707e5cba99df387 P e1f9f638263945d7b85ff6c2c237c2f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:45:52.013186 22526 raft_consensus.cc:729] T 0f6083e823d545349707e5cba99df387 P e1f9f638263945d7b85ff6c2c237c2f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1f9f638263945d7b85ff6c2c237c2f1, State: Initialized, Role: FOLLOWER
I20260620 18:45:52.013739 22526 consensus_queue.cc:260] T 0f6083e823d545349707e5cba99df387 P e1f9f638263945d7b85ff6c2c237c2f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d8f23504b8141be866b091a88bd6ace" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 34311 } } peers { permanent_uuid: "6f8524c2278f4cd29d9f88ba41228cc6" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 43087 } } peers { permanent_uuid: "e1f9f638263945d7b85ff6c2c237c2f1" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 41609 } }
I20260620 18:45:52.014854 22246 heartbeater.cc:503] Master 127.5.225.190:45289 was elected leader, sending a full tablet report...
I20260620 18:45:52.014621 22525 raft_consensus.cc:348] T 0f6083e823d545349707e5cba99df387 P 7d8f23504b8141be866b091a88bd6ace [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d8f23504b8141be866b091a88bd6ace" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 34311 } } peers { permanent_uuid: "6f8524c2278f4cd29d9f88ba41228cc6" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 43087 } } peers { permanent_uuid: "e1f9f638263945d7b85ff6c2c237c2f1" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 41609 } }
I20260620 18:45:52.015017 22525 raft_consensus.cc:374] T 0f6083e823d545349707e5cba99df387 P 7d8f23504b8141be866b091a88bd6ace [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:45:52.015110 22525 raft_consensus.cc:729] T 0f6083e823d545349707e5cba99df387 P 7d8f23504b8141be866b091a88bd6ace [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d8f23504b8141be866b091a88bd6ace, State: Initialized, Role: FOLLOWER
I20260620 18:45:52.015522 22526 ts_tablet_manager.cc:1428] T 0f6083e823d545349707e5cba99df387 P e1f9f638263945d7b85ff6c2c237c2f1: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260620 18:45:52.015811 22525 consensus_queue.cc:260] T 0f6083e823d545349707e5cba99df387 P 7d8f23504b8141be866b091a88bd6ace [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d8f23504b8141be866b091a88bd6ace" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 34311 } } peers { permanent_uuid: "6f8524c2278f4cd29d9f88ba41228cc6" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 43087 } } peers { permanent_uuid: "e1f9f638263945d7b85ff6c2c237c2f1" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 41609 } }
I20260620 18:45:52.017041 22378 heartbeater.cc:503] Master 127.5.225.190:45289 was elected leader, sending a full tablet report...
I20260620 18:45:52.017480 22525 ts_tablet_manager.cc:1428] T 0f6083e823d545349707e5cba99df387 P 7d8f23504b8141be866b091a88bd6ace: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260620 18:45:52.054118 22533 raft_consensus.cc:482] T 0f6083e823d545349707e5cba99df387 P 7d8f23504b8141be866b091a88bd6ace [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:45:52.054530 22533 raft_consensus.cc:504] T 0f6083e823d545349707e5cba99df387 P 7d8f23504b8141be866b091a88bd6ace [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d8f23504b8141be866b091a88bd6ace" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 34311 } } peers { permanent_uuid: "6f8524c2278f4cd29d9f88ba41228cc6" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 43087 } } peers { permanent_uuid: "e1f9f638263945d7b85ff6c2c237c2f1" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 41609 } }
I20260620 18:45:52.056591 22533 leader_election.cc:288] T 0f6083e823d545349707e5cba99df387 P 7d8f23504b8141be866b091a88bd6ace [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f8524c2278f4cd29d9f88ba41228cc6 (127.5.225.131:43087), e1f9f638263945d7b85ff6c2c237c2f1 (127.5.225.129:41609)
I20260620 18:45:52.062903 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:52.056830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:34527 (local address 127.5.225.131:43087)
0620 18:45:52.057079 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:52.057088 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:52.057754 (+ 666us) server_negotiation.cc:407] Connection header received
0620 18:45:52.057990 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:52.057994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:52.058054 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:52.058150 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:45:52.058698 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:52.059408 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:52.061182 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:52.061967 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:52.062145 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:52.062246 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:52.062527 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 18:45:52.062650 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 18:45:52.063045 22534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:52.056802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.131:43087 (local address 127.5.225.130:34527)
0620 18:45:52.057644 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:52.057660 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:52.057837 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:52.058185 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:52.058190 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:52.058215 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:45:52.058556 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:52.058564 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:52.059543 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:52.059548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:52.061032 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:52.061043 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:52.062211 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:52.062218 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:52.062369 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:52.062392 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 18:45:52.062501 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 18:45:52.062621 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":60,"thread_start_us":509,"threads_started":1}
I20260620 18:45:52.063910 22522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:52.057171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:41703 (local address 127.5.225.129:41609)
0620 18:45:52.057427 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:52.057437 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:52.058120 (+ 683us) server_negotiation.cc:407] Connection header received
0620 18:45:52.058396 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:52.058401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:52.058465 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:52.058569 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:45:52.059142 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:52.059926 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:52.061774 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:52.062703 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:52.062890 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:52.062976 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:52.063505 (+ 529us) server_negotiation.cc:299] Negotiation successful
0620 18:45:52.063636 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 18:45:52.064127 22535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:52.056843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:41609 (local address 127.5.225.130:41703)
0620 18:45:52.057996 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:52.058011 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:52.058210 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:52.058587 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:52.058592 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:52.058617 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:45:52.058982 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:52.058991 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:52.060068 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:52.060078 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:52.061614 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:52.061626 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:52.062943 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:52.062950 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:52.063323 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:52.063349 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 18:45:52.063466 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 18:45:52.063604 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1041,"mutex_wait_us":7,"thread_start_us":839,"threads_started":1}
I20260620 18:45:52.063944 22464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f6083e823d545349707e5cba99df387" candidate_uuid: "7d8f23504b8141be866b091a88bd6ace" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f8524c2278f4cd29d9f88ba41228cc6" is_pre_election: true
I20260620 18:45:52.064522 22464 raft_consensus.cc:2454] T 0f6083e823d545349707e5cba99df387 P 6f8524c2278f4cd29d9f88ba41228cc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d8f23504b8141be866b091a88bd6ace in term 0.
I20260620 18:45:52.065161 22269 leader_election.cc:302] T 0f6083e823d545349707e5cba99df387 P 7d8f23504b8141be866b091a88bd6ace [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f8524c2278f4cd29d9f88ba41228cc6, 7d8f23504b8141be866b091a88bd6ace; no voters:
I20260620 18:45:52.065138 22201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f6083e823d545349707e5cba99df387" candidate_uuid: "7d8f23504b8141be866b091a88bd6ace" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1f9f638263945d7b85ff6c2c237c2f1" is_pre_election: true
I20260620 18:45:52.065601 22533 raft_consensus.cc:2781] T 0f6083e823d545349707e5cba99df387 P 7d8f23504b8141be866b091a88bd6ace [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:45:52.065717 22533 raft_consensus.cc:482] T 0f6083e823d545349707e5cba99df387 P 7d8f23504b8141be866b091a88bd6ace [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:45:52.065725 22201 raft_consensus.cc:2454] T 0f6083e823d545349707e5cba99df387 P e1f9f638263945d7b85ff6c2c237c2f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d8f23504b8141be866b091a88bd6ace in term 0.
I20260620 18:45:52.065855 22533 raft_consensus.cc:3037] T 0f6083e823d545349707e5cba99df387 P 7d8f23504b8141be866b091a88bd6ace [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:45:52.068097 22533 raft_consensus.cc:504] T 0f6083e823d545349707e5cba99df387 P 7d8f23504b8141be866b091a88bd6ace [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d8f23504b8141be866b091a88bd6ace" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 34311 } } peers { permanent_uuid: "6f8524c2278f4cd29d9f88ba41228cc6" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 43087 } } peers { permanent_uuid: "e1f9f638263945d7b85ff6c2c237c2f1" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 41609 } }
I20260620 18:45:52.068727 22533 leader_election.cc:288] T 0f6083e823d545349707e5cba99df387 P 7d8f23504b8141be866b091a88bd6ace [CANDIDATE]: Term 1 election: Requested vote from peers 6f8524c2278f4cd29d9f88ba41228cc6 (127.5.225.131:43087), e1f9f638263945d7b85ff6c2c237c2f1 (127.5.225.129:41609)
I20260620 18:45:52.069270 22464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f6083e823d545349707e5cba99df387" candidate_uuid: "7d8f23504b8141be866b091a88bd6ace" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f8524c2278f4cd29d9f88ba41228cc6"
I20260620 18:45:52.069346 22201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f6083e823d545349707e5cba99df387" candidate_uuid: "7d8f23504b8141be866b091a88bd6ace" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1f9f638263945d7b85ff6c2c237c2f1"
I20260620 18:45:52.069481 22464 raft_consensus.cc:3037] T 0f6083e823d545349707e5cba99df387 P 6f8524c2278f4cd29d9f88ba41228cc6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:45:52.069532 22201 raft_consensus.cc:3037] T 0f6083e823d545349707e5cba99df387 P e1f9f638263945d7b85ff6c2c237c2f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:45:52.072744 22201 raft_consensus.cc:2454] T 0f6083e823d545349707e5cba99df387 P e1f9f638263945d7b85ff6c2c237c2f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d8f23504b8141be866b091a88bd6ace in term 1.
I20260620 18:45:52.072744 22464 raft_consensus.cc:2454] T 0f6083e823d545349707e5cba99df387 P 6f8524c2278f4cd29d9f88ba41228cc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d8f23504b8141be866b091a88bd6ace in term 1.
I20260620 18:45:52.073112 22269 leader_election.cc:302] T 0f6083e823d545349707e5cba99df387 P 7d8f23504b8141be866b091a88bd6ace [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f8524c2278f4cd29d9f88ba41228cc6, 7d8f23504b8141be866b091a88bd6ace; no voters:
I20260620 18:45:52.073402 22533 raft_consensus.cc:2781] T 0f6083e823d545349707e5cba99df387 P 7d8f23504b8141be866b091a88bd6ace [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:45:52.073836 22533 raft_consensus.cc:686] T 0f6083e823d545349707e5cba99df387 P 7d8f23504b8141be866b091a88bd6ace [term 1 LEADER]: Becoming Leader. State: Replica: 7d8f23504b8141be866b091a88bd6ace, State: Running, Role: LEADER
I20260620 18:45:52.074333 22533 consensus_queue.cc:237] T 0f6083e823d545349707e5cba99df387 P 7d8f23504b8141be866b091a88bd6ace [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d8f23504b8141be866b091a88bd6ace" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 34311 } } peers { permanent_uuid: "6f8524c2278f4cd29d9f88ba41228cc6" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 43087 } } peers { permanent_uuid: "e1f9f638263945d7b85ff6c2c237c2f1" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 41609 } }
I20260620 18:45:52.078778 22060 catalog_manager.cc:5495] T 0f6083e823d545349707e5cba99df387 P 7d8f23504b8141be866b091a88bd6ace reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d8f23504b8141be866b091a88bd6ace (127.5.225.130). New cstate: current_term: 1 leader_uuid: "7d8f23504b8141be866b091a88bd6ace" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d8f23504b8141be866b091a88bd6ace" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 34311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f8524c2278f4cd29d9f88ba41228cc6" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 43087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1f9f638263945d7b85ff6c2c237c2f1" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 41609 } health_report { overall_health: UNKNOWN } } }
I20260620 18:45:52.123369 22520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:52.117587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59168 (local address 127.5.225.130:34311)
0620 18:45:52.117770 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:52.117775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:52.117792 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:45:52.117987 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:52.117991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:52.118077 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:52.118282 (+ 205us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:45:52.118720 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:52.119464 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:52.120970 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:52.121452 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:52.121569 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:52.122087 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:45:52.122697 (+ 610us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:45:52.122810 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:52.123011 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 18:45:52.123083 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:45:52.131970 22464 raft_consensus.cc:1261] T 0f6083e823d545349707e5cba99df387 P 6f8524c2278f4cd29d9f88ba41228cc6 [term 1 FOLLOWER]: Refusing update from remote peer 7d8f23504b8141be866b091a88bd6ace: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:45:52.131974 22201 raft_consensus.cc:1261] T 0f6083e823d545349707e5cba99df387 P e1f9f638263945d7b85ff6c2c237c2f1 [term 1 FOLLOWER]: Refusing update from remote peer 7d8f23504b8141be866b091a88bd6ace: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260620 18:45:52.133076 22510 tablet.cc:2369] T 0f6083e823d545349707e5cba99df387 P 6f8524c2278f4cd29d9f88ba41228cc6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:45:52.133118 22536 consensus_queue.cc:1035] T 0f6083e823d545349707e5cba99df387 P 7d8f23504b8141be866b091a88bd6ace [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1f9f638263945d7b85ff6c2c237c2f1" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 41609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:45:52.133951 22533 consensus_queue.cc:1035] T 0f6083e823d545349707e5cba99df387 P 7d8f23504b8141be866b091a88bd6ace [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f8524c2278f4cd29d9f88ba41228cc6" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 43087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:45:52.162395 22060 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:45:52.162688 22060 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:45:52.166460 22060 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049: Sending DeleteTablet for 3 replicas of tablet 0f6083e823d545349707e5cba99df387
I20260620 18:45:52.167737 22444 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f6083e823d545349707e5cba99df387 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:45:52 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:34804
I20260620 18:45:52.167790 22181 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f6083e823d545349707e5cba99df387 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:45:52 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:50290
I20260620 18:45:52.167745 22313 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f6083e823d545349707e5cba99df387 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:45:52 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:59160
I20260620 18:45:52.169186 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 22098
I20260620 18:45:52.180335 22550 tablet_replica.cc:330] T 0f6083e823d545349707e5cba99df387 P 6f8524c2278f4cd29d9f88ba41228cc6: stopping tablet replica
I20260620 18:45:52.180389 22549 tablet_replica.cc:330] T 0f6083e823d545349707e5cba99df387 P e1f9f638263945d7b85ff6c2c237c2f1: stopping tablet replica
I20260620 18:45:52.180861 22549 raft_consensus.cc:2229] T 0f6083e823d545349707e5cba99df387 P e1f9f638263945d7b85ff6c2c237c2f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:45:52.180851 22550 raft_consensus.cc:2229] T 0f6083e823d545349707e5cba99df387 P 6f8524c2278f4cd29d9f88ba41228cc6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:45:52.181408 22549 raft_consensus.cc:2258] T 0f6083e823d545349707e5cba99df387 P e1f9f638263945d7b85ff6c2c237c2f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:45:52.181412 22550 raft_consensus.cc:2258] T 0f6083e823d545349707e5cba99df387 P 6f8524c2278f4cd29d9f88ba41228cc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:45:52.185127 22550 ts_tablet_manager.cc:1905] T 0f6083e823d545349707e5cba99df387 P 6f8524c2278f4cd29d9f88ba41228cc6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:45:52.186717 22549 ts_tablet_manager.cc:1905] T 0f6083e823d545349707e5cba99df387 P e1f9f638263945d7b85ff6c2c237c2f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:45:52.191408 22522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:52.170827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50292 (local address 127.5.225.129:41609)
0620 18:45:52.171011 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:52.171017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:52.171035 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:45:52.171096 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:52.171100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:52.171204 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:52.171307 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:52.173491 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:52.174691 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:52.176739 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:52.177424 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:52.177551 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:52.182773 (+ 5222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:52.182790 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:52.182810 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:52.182860 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:52.187070 (+ 4210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:52.187984 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:52.187994 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:52.187998 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:52.188154 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:52.188604 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:52.188609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:52.188613 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:52.189158 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:52.189486 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:52.190750 (+ 1264us) server_negotiation.cc:299] Negotiation successful
0620 18:45:52.191007 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:45:52.191846 22551 tablet_replica.cc:330] T 0f6083e823d545349707e5cba99df387 P 7d8f23504b8141be866b091a88bd6ace: stopping tablet replica
I20260620 18:45:52.192363 22241 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:52.192342 22551 raft_consensus.cc:2229] T 0f6083e823d545349707e5cba99df387 P 7d8f23504b8141be866b091a88bd6ace [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:45:52.192464 22550 ts_tablet_manager.cc:1918] T 0f6083e823d545349707e5cba99df387 P 6f8524c2278f4cd29d9f88ba41228cc6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:45:52.192603 22550 log.cc:1196] T 0f6083e823d545349707e5cba99df387 P 6f8524c2278f4cd29d9f88ba41228cc6: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/wal/wals/0f6083e823d545349707e5cba99df387
I20260620 18:45:52.193424 22550 ts_tablet_manager.cc:1939] T 0f6083e823d545349707e5cba99df387 P 6f8524c2278f4cd29d9f88ba41228cc6: Deleting consensus metadata
I20260620 18:45:52.193408 22551 raft_consensus.cc:2258] T 0f6083e823d545349707e5cba99df387 P 7d8f23504b8141be866b091a88bd6ace [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:45:52.195402 22048 catalog_manager.cc:4841] TS 6f8524c2278f4cd29d9f88ba41228cc6 (127.5.225.131:43087): tablet 0f6083e823d545349707e5cba99df387 (table test-table [id=83c562aa843c4ba3aaa024fec9222496]) successfully deleted
I20260620 18:45:52.199116 22551 ts_tablet_manager.cc:1905] T 0f6083e823d545349707e5cba99df387 P 7d8f23504b8141be866b091a88bd6ace: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:45:52.206681 22551 ts_tablet_manager.cc:1918] T 0f6083e823d545349707e5cba99df387 P 7d8f23504b8141be866b091a88bd6ace: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:45:52.207021 22551 log.cc:1196] T 0f6083e823d545349707e5cba99df387 P 7d8f23504b8141be866b091a88bd6ace: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/wal/wals/0f6083e823d545349707e5cba99df387
I20260620 18:45:52.207639 22551 ts_tablet_manager.cc:1939] T 0f6083e823d545349707e5cba99df387 P 7d8f23504b8141be866b091a88bd6ace: Deleting consensus metadata
I20260620 18:45:52.209539 22048 catalog_manager.cc:4841] TS 7d8f23504b8141be866b091a88bd6ace (127.5.225.130:34311): tablet 0f6083e823d545349707e5cba99df387 (table test-table [id=83c562aa843c4ba3aaa024fec9222496]) successfully deleted
I20260620 18:45:52.287009 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 22098
I20260620 18:45:52.301898 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 22250
W20260620 18:45:52.301867 22047 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.225.129:41609 (error 108)
I20260620 18:45:52.303848 22524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:52.303381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:41609 (local address 127.0.0.1:50300)
0620 18:45:52.303536 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:52.303657 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.129:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 18:45:52.304222 22047 catalog_manager.cc:4584] TS e1f9f638263945d7b85ff6c2c237c2f1 (127.5.225.129:41609): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0f6083e823d545349707e5cba99df387: Network error: Client connection negotiation failed: client connection to 127.5.225.129:41609: connect: Connection refused (error 111)
I20260620 18:45:52.317452 22520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:52.303113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59172 (local address 127.5.225.130:34311)
0620 18:45:52.303280 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:52.303284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:52.303632 (+ 348us) server_negotiation.cc:407] Connection header received
0620 18:45:52.303861 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:52.303869 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:52.303984 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:52.304120 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:52.305599 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:52.306561 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:52.307915 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:52.308368 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:52.308499 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:52.311210 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:52.311224 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:52.311239 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:52.311280 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:52.314657 (+ 3377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:52.315339 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:52.315354 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:52.315357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:52.315497 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:52.315893 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:52.315897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:52.315899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:52.316264 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:52.316588 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:52.316846 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 18:45:52.317085 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:45:52.318184 22373 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:52.358058 22524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:52.357493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:41609 (local address 127.0.0.1:50314)
0620 18:45:52.357732 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:52.357860 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.129:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 18:45:52.359179 22524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:52.358780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:41609 (local address 127.0.0.1:50320)
0620 18:45:52.358947 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:52.359035 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.129:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 18:45:52.419730 22524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:52.419143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:41609 (local address 127.0.0.1:50336)
0620 18:45:52.419431 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:52.419545 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.129:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260620 18:45:52.420923 22524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:52.420521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:41609 (local address 127.0.0.1:50352)
0620 18:45:52.420688 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:52.420773 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.129:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 18:45:52.432359 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 22250
I20260620 18:45:52.451757 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 22381
I20260620 18:45:52.466056 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:52.452462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34818 (local address 127.5.225.131:43087)
0620 18:45:52.452647 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:52.452653 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:52.452704 (+ 51us) server_negotiation.cc:407] Connection header received
0620 18:45:52.452935 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:52.452941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:52.453013 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:52.453120 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:52.454550 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:52.455303 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:52.456675 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:52.457093 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:52.457223 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:52.460079 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:52.460094 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:52.460109 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:52.460143 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:52.463127 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:52.463872 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:52.463879 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:52.463881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:52.464109 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:52.464493 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:52.464499 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:52.464502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:52.465036 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:52.465399 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:52.465632 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 18:45:52.465802 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:45:52.466707 22504 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:52.501789 22524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:52.501244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:41609 (local address 127.0.0.1:50358)
0620 18:45:52.501483 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:52.501617 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.129:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 18:45:52.503033 22524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:52.502568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:41609 (local address 127.0.0.1:50360)
0620 18:45:52.502798 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:52.502890 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.129:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260620 18:45:52.556643 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 22381
I20260620 18:45:52.574234 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 22028
I20260620 18:45:52.575515 22091 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:52.690869 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 22028
I20260620 18:46:12.713409 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:45289
--webserver_interface=127.5.225.190
--webserver_port=45995
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:35541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:45289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:46:13.046804 22557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:46:13.047139 22557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:46:13.047299 22557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:46:13.054534 22557 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 18:46:13.054666 22557 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 18:46:13.054759 22557 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 18:46:13.054854 22557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:46:13.054953 22557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:46:13.055081 22557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:46:13.055171 22557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:46:13.055249 22557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:46:13.065073 22557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:35541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:45289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:45289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=45995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:46:13.066418 22557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:46:13.067906 22557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:46:13.080214 22563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:13.080161 22565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:13.080159 22562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:46:13.081055 22557 server_base.cc:1034] running on GCE node
I20260620 18:46:13.081875 22557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:46:13.083427 22557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:46:13.084700 22557 hybrid_clock.cc:648] HybridClock initialized: now 1781981173084607 us; error 41 us; skew 500 ppm
I20260620 18:46:13.085050 22557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:46:13 dist-test-slave-k6lg krb5kdc[21478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981173, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:46:13.090939 22557 init.cc:377] Logged in from keytab as kudu/127.5.225.190@KRBTEST.COM (short username kudu)
I20260620 18:46:13.092478 22557 webserver.cc:460] Webserver started at http://127.5.225.190:45995/ using document root <none> and password file <none>
I20260620 18:46:13.093185 22557 fs_manager.cc:359] Metadata directory not provided
I20260620 18:46:13.093322 22557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:46:13.098793 22557 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:46:13.101598 22572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:13.102999 22557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:46:13.103205 22557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "0f9a644e0e0146b4a66ef256b1f7a049"
format_stamp: "Formatted at 2026-06-20 18:45:27 on dist-test-slave-k6lg"
server_key: "25f877fe481f72bdacc5bfa67cb852ac"
server_key_iv: "ca562b88a2576b4771e3a3fba1827c71"
server_key_version: "encryptionkey@0"
I20260620 18:46:13.104185 22557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:46:13.131855 22557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:46:13.135939 22557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:46:13.136241 22557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:46:13.154855 22557 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:45289
I20260620 18:46:13.154886 22623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:45289 every 8 connection(s)
I20260620 18:46:13.156443 22557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:46:13.163146 22624 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:46:13.166338 22624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049: Bootstrap starting.
I20260620 18:46:13.166515 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 22557
I20260620 18:46:13.167392 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.129:41609
--local_ip_for_outbound_sockets=127.5.225.129
--tserver_master_addrs=127.5.225.190:45289
--webserver_port=34483
--webserver_interface=127.5.225.129
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:35541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:46:13.189429 22624 log.cc:824] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049: Log is configured to *not* fsync() on all Append() calls
I20260620 18:46:13.205097 22624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:46:13.205685 22624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049: Bootstrap complete.
I20260620 18:46:13.215420 22624 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f9a644e0e0146b4a66ef256b1f7a049" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 45289 } }
I20260620 18:46:13.216323 22624 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f9a644e0e0146b4a66ef256b1f7a049, State: Initialized, Role: FOLLOWER
I20260620 18:46:13.216980 22624 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f9a644e0e0146b4a66ef256b1f7a049" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 45289 } }
I20260620 18:46:13.217222 22624 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:46:13.217358 22624 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:46:13.217537 22624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [term 2 FOLLOWER]: Advancing to term 3
I20260620 18:46:13.221300 22624 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f9a644e0e0146b4a66ef256b1f7a049" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 45289 } }
I20260620 18:46:13.221781 22624 leader_election.cc:302] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f9a644e0e0146b4a66ef256b1f7a049; no voters:
I20260620 18:46:13.222484 22624 leader_election.cc:288] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 18:46:13.222628 22628 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 18:46:13.223886 22628 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [term 3 LEADER]: Becoming Leader. State: Replica: 0f9a644e0e0146b4a66ef256b1f7a049, State: Running, Role: LEADER
I20260620 18:46:13.224434 22628 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f9a644e0e0146b4a66ef256b1f7a049" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 45289 } }
I20260620 18:46:13.224793 22624 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:46:13.227152 22630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f9a644e0e0146b4a66ef256b1f7a049. Latest consensus state: current_term: 3 leader_uuid: "0f9a644e0e0146b4a66ef256b1f7a049" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f9a644e0e0146b4a66ef256b1f7a049" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 45289 } } }
I20260620 18:46:13.227530 22630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [sys.catalog]: This master's current role is: LEADER
I20260620 18:46:13.227995 22629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0f9a644e0e0146b4a66ef256b1f7a049" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f9a644e0e0146b4a66ef256b1f7a049" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 45289 } } }
I20260620 18:46:13.229452 22629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049 [sys.catalog]: This master's current role is: LEADER
I20260620 18:46:13.230826 22637 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:46:13.241153 22637 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:46:13.242573 22637 catalog_manager.cc:1259] Loaded cluster ID: ad104ace8f224e259c912cbb453d1f6b
I20260620 18:46:13.242686 22637 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:46:13.247079 22637 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:46:13.248857 22637 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049: Loaded TSK: 0
I20260620 18:46:13.248996 22637 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0f9a644e0e0146b4a66ef256b1f7a049: Loaded TSK: 1
F20260620 18:46:13.250061 22637 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6a314629ad google::LogMessage::Fail() at ??:0
@ 0x7f6a3146588a google::LogMessage::SendToLog() at ??:0
@ 0x7f6a314624de google::LogMessage::Flush() at ??:0
@ 0x7f6a31463a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6a351c67d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f6a351b573e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f6a46e911c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f6a46e90783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f6a46f1c2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f6a46f1c0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f6a46f665e9 std::function<>::operator()() at ??:0
@ 0x7f6a46e94cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f6a46e92c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6a46f1a610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f6a46f1a439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f6a340ae345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6a340b191c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f6a340b1799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f6a3408ceaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6a373666db start_thread at ??:0
@ 0x7f6a302fc61f clone at ??:0
W20260620 18:46:13.550662 22626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:46:13.550938 22626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:46:13.551034 22626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:46:13.558398 22626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:46:13.558508 22626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:46:13.558640 22626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.129
I20260620 18:46:13.568480 22626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:35541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.129:41609
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.225.129
--webserver_port=34483
--tserver_master_addrs=127.5.225.190:45289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.129
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:46:13.569778 22626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:46:13.571162 22626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:46:13.583896 22672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:13.583858 22674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:13.583959 22671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:46:13.584609 22626 server_base.cc:1034] running on GCE node
I20260620 18:46:13.585152 22626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:46:13.586113 22626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:46:13.587359 22626 hybrid_clock.cc:648] HybridClock initialized: now 1781981173587313 us; error 32 us; skew 500 ppm
I20260620 18:46:13.587684 22626 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:46:13 dist-test-slave-k6lg krb5kdc[21478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981173, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:46:13.590782 22626 init.cc:377] Logged in from keytab as kudu/127.5.225.129@KRBTEST.COM (short username kudu)
I20260620 18:46:13.592135 22626 webserver.cc:460] Webserver started at http://127.5.225.129:34483/ using document root <none> and password file <none>
I20260620 18:46:13.592814 22626 fs_manager.cc:359] Metadata directory not provided
I20260620 18:46:13.592949 22626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:46:13.598021 22626 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:46:13.600850 22681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:13.602119 22626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:46:13.602363 22626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/wal
uuid: "e1f9f638263945d7b85ff6c2c237c2f1"
format_stamp: "Formatted at 2026-06-20 18:45:27 on dist-test-slave-k6lg"
server_key: "f0a4c2e6ed04336be52dda6992a19228"
server_key_iv: "ec7e732668ae09a39358427b02d70eb5"
server_key_version: "encryptionkey@0"
I20260620 18:46:13.603314 22626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:46:13.616662 22626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:46:13.620605 22626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:46:13.620879 22626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:46:13.621816 22626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:46:13.623664 22688 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260620 18:46:13.628795 22688 ts_tablet_manager.cc:1852] T 0f6083e823d545349707e5cba99df387 P e1f9f638263945d7b85ff6c2c237c2f1: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260620 18:46:13.629760 22688 ts_tablet_manager.cc:1905] T 0f6083e823d545349707e5cba99df387 P e1f9f638263945d7b85ff6c2c237c2f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:46:13.636716 22688 ts_tablet_manager.cc:1918] T 0f6083e823d545349707e5cba99df387 P e1f9f638263945d7b85ff6c2c237c2f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:46:13.636904 22688 log.cc:1196] T 0f6083e823d545349707e5cba99df387 P e1f9f638263945d7b85ff6c2c237c2f1: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/wal/wals/0f6083e823d545349707e5cba99df387
I20260620 18:46:13.637370 22688 ts_tablet_manager.cc:1939] T 0f6083e823d545349707e5cba99df387 P e1f9f638263945d7b85ff6c2c237c2f1: Deleting consensus metadata
I20260620 18:46:13.637743 22626 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260620 18:46:13.637881 22626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.001s sys 0.000s
I20260620 18:46:13.638020 22626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:46:13.638113 22626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:13.672124 22626 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.129:41609
I20260620 18:46:13.672180 22794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.129:41609 every 8 connection(s)
I20260620 18:46:13.673535 22626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
I20260620 18:46:13.676216 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 22626
I20260620 18:46:13.676956 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.130:34311
--local_ip_for_outbound_sockets=127.5.225.130
--tserver_master_addrs=127.5.225.190:45289
--webserver_port=43493
--webserver_interface=127.5.225.130
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:35541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:46:13.679126 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:13.677066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.129:43979)
0620 18:46:13.677873 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:13.678231 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":324,"threads_started":1}
W20260620 18:46:13.679716 22795 heartbeater.cc:644] Failed to heartbeat to 127.5.225.190:45289 (0 consecutive failures): Network error: Failed to ping master at 127.5.225.190:45289: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
I20260620 18:46:13.680615 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:13.680307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.129:33105)
0620 18:46:13.680451 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:13.680517 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 18:46:13.681545 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:13.681254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.129:54397)
0620 18:46:13.681371 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:13.681437 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 18:46:13.681818 22795 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:46:14.007855 22798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:46:14.008132 22798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:46:14.008227 22798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:46:14.015554 22798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:46:14.015671 22798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:46:14.015782 22798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.130
I20260620 18:46:14.025480 22798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:35541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.130:34311
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.225.130
--webserver_port=43493
--tserver_master_addrs=127.5.225.190:45289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.130
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:46:14.026841 22798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:46:14.028160 22798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:46:14.041365 22803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:14.041437 22804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:14.041406 22806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:46:14.042483 22798 server_base.cc:1034] running on GCE node
I20260620 18:46:14.043205 22798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:46:14.044140 22798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:46:14.045342 22798 hybrid_clock.cc:648] HybridClock initialized: now 1781981174045279 us; error 53 us; skew 500 ppm
I20260620 18:46:14.045670 22798 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:46:14 dist-test-slave-k6lg krb5kdc[21478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981174, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:46:14.048775 22798 init.cc:377] Logged in from keytab as kudu/127.5.225.130@KRBTEST.COM (short username kudu)
I20260620 18:46:14.050320 22798 webserver.cc:460] Webserver started at http://127.5.225.130:43493/ using document root <none> and password file <none>
I20260620 18:46:14.050956 22798 fs_manager.cc:359] Metadata directory not provided
I20260620 18:46:14.051059 22798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:46:14.056233 22798 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 18:46:14.059242 22813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:14.060576 22798 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:46:14.060758 22798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/wal
uuid: "7d8f23504b8141be866b091a88bd6ace"
format_stamp: "Formatted at 2026-06-20 18:45:28 on dist-test-slave-k6lg"
server_key: "1431d5002b2e2fba1bb57522ac2fef17"
server_key_iv: "e6d97a8d9a471bb66ed86867698a539d"
server_key_version: "encryptionkey@0"
I20260620 18:46:14.061749 22798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:46:14.080935 22798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:46:14.085029 22798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:46:14.085300 22798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:46:14.086333 22798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:46:14.087819 22798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:46:14.087937 22798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:14.088054 22798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:46:14.088145 22798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:14.123495 22798 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.130:34311
I20260620 18:46:14.123545 22925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.130:34311 every 8 connection(s)
I20260620 18:46:14.124955 22798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
I20260620 18:46:14.129812 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:14.128037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.130:56789)
0620 18:46:14.128766 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:14.128991 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":275,"threads_started":1}
W20260620 18:46:14.130532 22926 heartbeater.cc:644] Failed to heartbeat to 127.5.225.190:45289 (0 consecutive failures): Network error: Failed to ping master at 127.5.225.190:45289: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
I20260620 18:46:14.131310 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:14.131010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.130:39133)
0620 18:46:14.131147 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:14.131211 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 18:46:14.132057 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 22798
I20260620 18:46:14.132280 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:14.131959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.130:36775)
0620 18:46:14.132083 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:14.132148 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 18:46:14.132535 22926 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 18:46:14.132784 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.131:43087
--local_ip_for_outbound_sockets=127.5.225.131
--tserver_master_addrs=127.5.225.190:45289
--webserver_port=36111
--webserver_interface=127.5.225.131
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:35541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:46:14.450904 22929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:46:14.451180 22929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:46:14.451274 22929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:46:14.458278 22929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:46:14.458379 22929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:46:14.458488 22929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.131
I20260620 18:46:14.468168 22929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:35541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.131:43087
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.225.131
--webserver_port=36111
--tserver_master_addrs=127.5.225.190:45289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.131
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:46:14.469328 22929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:46:14.470597 22929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:46:14.483271 22934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:14.483290 22935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:14.483596 22937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:46:14.484601 22929 server_base.cc:1034] running on GCE node
I20260620 18:46:14.485220 22929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:46:14.486124 22929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:46:14.487375 22929 hybrid_clock.cc:648] HybridClock initialized: now 1781981174487328 us; error 42 us; skew 500 ppm
I20260620 18:46:14.487737 22929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:46:14 dist-test-slave-k6lg krb5kdc[21478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981174, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:46:14.490829 22929 init.cc:377] Logged in from keytab as kudu/127.5.225.131@KRBTEST.COM (short username kudu)
I20260620 18:46:14.492215 22929 webserver.cc:460] Webserver started at http://127.5.225.131:36111/ using document root <none> and password file <none>
I20260620 18:46:14.492869 22929 fs_manager.cc:359] Metadata directory not provided
I20260620 18:46:14.493014 22929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:46:14.498049 22929 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:46:14.500530 22944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:14.501806 22929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:46:14.501971 22929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/wal
uuid: "6f8524c2278f4cd29d9f88ba41228cc6"
format_stamp: "Formatted at 2026-06-20 18:45:28 on dist-test-slave-k6lg"
server_key: "ef8f8c8c7e070ddf046e4f98e6e8321c"
server_key_iv: "9f0c21d2af9a17cd04dda8d1b6efe893"
server_key_version: "encryptionkey@0"
I20260620 18:46:14.502995 22929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:46:14.516884 22929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:46:14.521270 22929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:46:14.521493 22929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:46:14.522406 22929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:46:14.523653 22929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:46:14.523746 22929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:14.523818 22929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:46:14.523929 22929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:14.555505 22929 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.131:43087
I20260620 18:46:14.555579 23056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.131:43087 every 8 connection(s)
I20260620 18:46:14.557049 22929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
I20260620 18:46:14.561712 23059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:14.559989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.131:41927)
0620 18:46:14.560682 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:14.560910 (+ 228us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":259,"threads_started":1}
W20260620 18:46:14.562471 23057 heartbeater.cc:644] Failed to heartbeat to 127.5.225.190:45289 (0 consecutive failures): Network error: Failed to ping master at 127.5.225.190:45289: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
I20260620 18:46:14.563359 23059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:14.563067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.131:51593)
0620 18:46:14.563203 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:14.563265 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 18:46:14.564149 23059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:14.563870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.131:50739)
0620 18:46:14.563988 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:14.564054 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 18:46:14.564393 23057 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 18:46:14.564992 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 22929
I20260620 18:46:14.683274 23061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:14.682471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.129:35667)
0620 18:46:14.682914 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:14.683020 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":235,"threads_started":1}
I20260620 18:46:15.134289 23062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:15.133212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.130:48147)
0620 18:46:15.133815 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:15.133958 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":306,"threads_started":1}
I20260620 18:46:15.565843 23063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:15.564893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.131:44189)
0620 18:46:15.565430 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:15.565533 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":278,"threads_started":1}
I20260620 18:46:15.684796 23064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:15.684125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.129:59273)
0620 18:46:15.684562 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:15.684643 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":238,"threads_started":1}
I20260620 18:46:16.136041 23065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:16.135346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.130:35681)
0620 18:46:16.135793 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:16.135876 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":227,"threads_started":1}
I20260620 18:46:16.567440 23066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:16.566715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.131:48273)
0620 18:46:16.567181 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:16.567265 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":220,"threads_started":1}
I20260620 18:46:16.686374 23067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:16.685604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.129:44603)
0620 18:46:16.686052 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:16.686135 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":227,"threads_started":1}
I20260620 18:46:17.137828 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:17.137120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.130:56825)
0620 18:46:17.137583 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:17.137662 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":242,"threads_started":1}
I20260620 18:46:17.569095 23069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:17.568358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.131:43925)
0620 18:46:17.568824 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:17.568911 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":254,"threads_started":1}
I20260620 18:46:17.687983 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:17.687218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.129:44495)
0620 18:46:17.687708 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:17.687800 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":256,"threads_started":1}
I20260620 18:46:18.139428 23071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:18.138694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.130:54155)
0620 18:46:18.139144 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:18.139231 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":203,"threads_started":1}
I20260620 18:46:18.570822 23072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:18.570042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.131:33739)
0620 18:46:18.570541 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:18.570625 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":274,"threads_started":1}
I20260620 18:46:18.689566 23073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:18.688870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.129:56741)
0620 18:46:18.689321 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:18.689402 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":235,"threads_started":1}
I20260620 18:46:19.141832 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:19.140481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.130:47577)
0620 18:46:19.141492 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:19.141627 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":911,"thread_start_us":314,"threads_started":1}
I20260620 18:46:19.572626 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:19.571888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.131:46509)
0620 18:46:19.572356 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:19.572433 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":256,"threads_started":1}
I20260620 18:46:19.691147 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:19.690471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.129:53585)
0620 18:46:19.690899 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:19.690991 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":217,"threads_started":1}
I20260620 18:46:20.143805 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:20.142850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.130:40469)
0620 18:46:20.143435 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:20.143547 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":299,"threads_started":1}
I20260620 18:46:20.574489 23078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:20.573662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.131:35131)
0620 18:46:20.574209 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:20.574314 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":275,"threads_started":1}
I20260620 18:46:20.692806 23079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:20.692032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.129:33369)
0620 18:46:20.692525 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:20.692604 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":228,"threads_started":1}
I20260620 18:46:21.146387 23080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:21.145298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.130:48039)
0620 18:46:21.145875 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:21.146013 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":264,"threads_started":1}
I20260620 18:46:21.576112 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:21.575346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.131:52107)
0620 18:46:21.575806 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:21.575892 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":208,"threads_started":1}
I20260620 18:46:21.694397 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:21.693701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.129:45651)
0620 18:46:21.694079 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:21.694213 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":195,"threads_started":1}
I20260620 18:46:22.148325 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:22.147624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.130:39073)
0620 18:46:22.148081 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:22.148164 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":232,"threads_started":1}
I20260620 18:46:22.577862 23084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:22.577112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.131:54261)
0620 18:46:22.577607 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:22.577697 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":249,"threads_started":1}
I20260620 18:46:22.696013 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:22.695243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.129:36143)
0620 18:46:22.695709 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:22.695803 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":232,"threads_started":1}
I20260620 18:46:23.149998 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:23.149254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.130:39671)
0620 18:46:23.149732 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:23.149813 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":224,"threads_started":1}
I20260620 18:46:23.579646 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:23.578856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.131:36217)
0620 18:46:23.579389 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:23.579471 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":269,"threads_started":1}
I20260620 18:46:23.697711 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:23.696954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.129:35659)
0620 18:46:23.697465 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:23.697544 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":294,"threads_started":1}
I20260620 18:46:24.151793 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:24.151044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.130:49305)
0620 18:46:24.151526 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:24.151618 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":252,"threads_started":1}
I20260620 18:46:24.581274 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:24.580602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.131:43929)
0620 18:46:24.581029 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:24.581111 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":234,"threads_started":1}
I20260620 18:46:24.699560 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:24.698754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.129:38679)
0620 18:46:24.699240 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:24.699325 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":272,"threads_started":1}
I20260620 18:46:25.153594 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:25.152781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.130:46529)
0620 18:46:25.153274 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:25.153365 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":271,"threads_started":1}
I20260620 18:46:25.583113 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:25.582287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.131:60415)
0620 18:46:25.582829 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:25.582926 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":324,"threads_started":1}
I20260620 18:46:25.701159 23094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:25.700491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.129:39333)
0620 18:46:25.700915 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:25.701005 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":216,"threads_started":1}
I20260620 18:46:26.155385 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:26.154674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.130:60445)
0620 18:46:26.155144 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:26.155229 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":245,"threads_started":1}
I20260620 18:46:26.584700 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:26.584032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.131:45141)
0620 18:46:26.584453 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:26.584535 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":216,"threads_started":1}
I20260620 18:46:26.702852 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:26.702050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.129:60529)
0620 18:46:26.702558 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:26.702654 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":267,"threads_started":1}
I20260620 18:46:27.157282 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:27.156418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.130:49901)
0620 18:46:27.156996 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:27.157088 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":327,"threads_started":1}
I20260620 18:46:27.586551 23099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:27.585675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.131:45697)
0620 18:46:27.586253 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:27.586359 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":266,"threads_started":1}
I20260620 18:46:27.704684 23100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:27.703757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.129:56557)
0620 18:46:27.704349 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:27.704457 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":315,"threads_started":1}
I20260620 18:46:28.159334 23101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:28.158291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.130:40127)
0620 18:46:28.158898 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:28.159071 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":286,"threads_started":1}
I20260620 18:46:28.588241 23102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:28.587527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.131:37289)
0620 18:46:28.587948 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:28.588061 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":209,"threads_started":1}
I20260620 18:46:28.706769 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:28.705892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.129:41341)
0620 18:46:28.706392 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:28.706486 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":224,"threads_started":1}
I20260620 18:46:29.161177 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:29.160429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.130:48337)
0620 18:46:29.160852 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:29.160985 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":220,"threads_started":1}
I20260620 18:46:29.568879 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 22626
I20260620 18:46:29.587262 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:29.570021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50076 (local address 127.5.225.129:41609)
0620 18:46:29.570624 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:29.570643 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:29.570877 (+ 234us) server_negotiation.cc:407] Connection header received
0620 18:46:29.572708 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:29.572718 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:29.573161 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:29.573519 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:29.575134 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:29.576382 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:29.577706 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:29.578415 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:29.578615 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:29.581146 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:29.581153 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:29.581168 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:46:29.581203 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:29.584042 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:29.584746 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:29.584753 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:29.584759 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:29.584916 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:29.585311 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:29.585317 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:29.585320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:29.585817 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:29.586141 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:29.586566 (+ 425us) server_negotiation.cc:299] Negotiation successful
0620 18:46:29.586984 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":302,"threads_started":1}
I20260620 18:46:29.588712 22790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:46:29.590448 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:29.589486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45289 (local address 127.5.225.131:49797)
0620 18:46:29.590058 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:29.590213 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.190:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":281,"threads_started":1}
I20260620 18:46:29.676297 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 22626
I20260620 18:46:29.694017 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 22798
I20260620 18:46:29.711910 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:29.695052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45728 (local address 127.5.225.130:34311)
0620 18:46:29.695562 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:29.695577 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:29.695614 (+ 37us) server_negotiation.cc:407] Connection header received
0620 18:46:29.697460 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:29.697475 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:29.698076 (+ 601us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:29.698509 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:29.699893 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:29.701141 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:29.702563 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:29.703234 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:29.703368 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:29.705917 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:29.705924 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:29.705939 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:46:29.705975 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:29.708862 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:29.709458 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:29.709465 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:29.709470 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:29.709642 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:29.709979 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:29.709983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:29.709984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:29.710634 (+ 650us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:29.710866 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:29.711215 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 18:46:29.711631 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":234,"threads_started":1}
I20260620 18:46:29.713245 22921 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:46:29.801136 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 22798
I20260620 18:46:29.818295 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 22929
I20260620 18:46:29.836098 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:29.819310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44742 (local address 127.5.225.131:43087)
0620 18:46:29.820008 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:29.820022 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:29.820064 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:46:29.821680 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:29.821690 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:29.822119 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:29.822534 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:29.823842 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:29.825090 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:29.826483 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:29.827156 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:29.827291 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:29.829739 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:29.829747 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:29.829761 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:46:29.829800 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:29.832885 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:29.833486 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:29.833492 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:29.833498 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:29.833655 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:29.834116 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:29.834119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:29.834121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:29.834687 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:29.834908 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:29.835317 (+ 409us) server_negotiation.cc:299] Negotiation successful
0620 18:46:29.835760 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":421,"threads_started":1}
I20260620 18:46:29.837463 23052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:46:29.919744 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 22929
2026-06-20T18:46:29Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64289 ms)
[----------] 30 tests from SecurityITest (295412 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:46:30 dist-test-slave-k6lg krb5kdc[23114](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:46:30 dist-test-slave-k6lg krb5kdc[23114](info): set up 2 sockets
Jun 20 18:46:30 dist-test-slave-k6lg krb5kdc[23114](info): commencing operation
krb5kdc: starting...
W20260620 18:46:31.022552 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:46:31 dist-test-slave-k6lg krb5kdc[23114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:46:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:46:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
I20260620 18:46:31.265841 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:37333
--webserver_interface=127.5.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:39001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:37333
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:46:31.593801 23127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:46:31.594081 23127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:46:31.594303 23127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:46:31.601454 23127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:46:31.601583 23127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:46:31.601680 23127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:46:31.601784 23127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:46:31.601866 23127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:46:31.611749 23127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:39001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:37333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:37333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:46:31.613111 23127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:46:31.614473 23127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:46:31.626505 23132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:31.626467 23135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:31.626473 23133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:46:31.627334 23127 server_base.cc:1034] running on GCE node
I20260620 18:46:31.628146 23127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:46:31.629505 23127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:46:31.630720 23127 hybrid_clock.cc:648] HybridClock initialized: now 1781981191630668 us; error 39 us; skew 500 ppm
I20260620 18:46:31.631119 23127 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:46:31 dist-test-slave-k6lg krb5kdc[23114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981191, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:46:31.637557 23127 init.cc:377] Logged in from keytab as kudu/127.5.225.190@KRBTEST.COM (short username kudu)
I20260620 18:46:31.639070 23127 webserver.cc:460] Webserver started at http://127.5.225.190:34911/ using document root <none> and password file <none>
I20260620 18:46:31.639753 23127 fs_manager.cc:359] Metadata directory not provided
I20260620 18:46:31.639887 23127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:46:31.640218 23127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:46:31.643217 23127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "b00294c8658947519fc4c046906ed089"
format_stamp: "Formatted at 2026-06-20 18:46:31 on dist-test-slave-k6lg"
server_key: "d59a0a0d629e3448c4fab55e415b8cad"
server_key_iv: "6d1fbcbd50f4824dfd83772eb22cf8ae"
server_key_version: "encryptionkey@0"
I20260620 18:46:31.643961 23127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "b00294c8658947519fc4c046906ed089"
format_stamp: "Formatted at 2026-06-20 18:46:31 on dist-test-slave-k6lg"
server_key: "d59a0a0d629e3448c4fab55e415b8cad"
server_key_iv: "6d1fbcbd50f4824dfd83772eb22cf8ae"
server_key_version: "encryptionkey@0"
I20260620 18:46:31.649087 23127 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:46:31.652622 23142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:31.654039 23127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 18:46:31.654265 23127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "b00294c8658947519fc4c046906ed089"
format_stamp: "Formatted at 2026-06-20 18:46:31 on dist-test-slave-k6lg"
server_key: "d59a0a0d629e3448c4fab55e415b8cad"
server_key_iv: "6d1fbcbd50f4824dfd83772eb22cf8ae"
server_key_version: "encryptionkey@0"
I20260620 18:46:31.654477 23127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:46:31.668752 23127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:46:31.672827 23127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:46:31.673148 23127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:46:31.692021 23127 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:37333
I20260620 18:46:31.692045 23193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:37333 every 8 connection(s)
I20260620 18:46:31.693406 23127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:46:31.697093 23194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:46:31.699162 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 23127
I20260620 18:46:31.699379 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:46:31.700320 6022 external_mini_cluster.cc:1428] Setting key ffb0202748b41e62eed09f746b71a687
I20260620 18:46:31.712340 23194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b00294c8658947519fc4c046906ed089: Bootstrap starting.
I20260620 18:46:31.715586 23194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b00294c8658947519fc4c046906ed089: Neither blocks nor log segments found. Creating new log.
I20260620 18:46:31.716598 23194 log.cc:824] T 00000000000000000000000000000000 P b00294c8658947519fc4c046906ed089: Log is configured to *not* fsync() on all Append() calls
I20260620 18:46:31.719616 23194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b00294c8658947519fc4c046906ed089: No bootstrap required, opened a new log
Jun 20 18:46:31 dist-test-slave-k6lg krb5kdc[23114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:46:31.729074 23194 raft_consensus.cc:348] T 00000000000000000000000000000000 P b00294c8658947519fc4c046906ed089 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00294c8658947519fc4c046906ed089" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 37333 } }
I20260620 18:46:31.729559 23194 raft_consensus.cc:374] T 00000000000000000000000000000000 P b00294c8658947519fc4c046906ed089 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:46:31.729679 23194 raft_consensus.cc:729] T 00000000000000000000000000000000 P b00294c8658947519fc4c046906ed089 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00294c8658947519fc4c046906ed089, State: Initialized, Role: FOLLOWER
I20260620 18:46:31.730424 23194 consensus_queue.cc:260] T 00000000000000000000000000000000 P b00294c8658947519fc4c046906ed089 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00294c8658947519fc4c046906ed089" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 37333 } }
I20260620 18:46:31.730710 23194 raft_consensus.cc:388] T 00000000000000000000000000000000 P b00294c8658947519fc4c046906ed089 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:46:31.730860 23194 raft_consensus.cc:482] T 00000000000000000000000000000000 P b00294c8658947519fc4c046906ed089 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:46:31.731046 23194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b00294c8658947519fc4c046906ed089 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:46:31.731371 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:31.702562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40466 (local address 127.5.225.190:37333)
0620 18:46:31.709631 (+ 7069us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:31.709652 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:31.709720 (+ 68us) server_negotiation.cc:407] Connection header received
0620 18:46:31.711662 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:31.711688 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:31.712145 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:31.712577 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:31.714105 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:31.715462 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:31.716900 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:31.718087 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:31.718324 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:31.723061 (+ 4737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:31.723070 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:31.723085 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:46:31.723128 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:31.726649 (+ 3521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:31.727550 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:31.727556 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:31.727566 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:31.727738 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:31.728186 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:31.728189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:31.728191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:31.728730 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:31.728953 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:31.729436 (+ 483us) server_negotiation.cc:299] Negotiation successful
0620 18:46:31.730000 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6748,"thread_start_us":6490,"threads_started":1}
I20260620 18:46:31.733445 23194 raft_consensus.cc:504] T 00000000000000000000000000000000 P b00294c8658947519fc4c046906ed089 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00294c8658947519fc4c046906ed089" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 37333 } }
I20260620 18:46:31.733989 23194 leader_election.cc:302] T 00000000000000000000000000000000 P b00294c8658947519fc4c046906ed089 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b00294c8658947519fc4c046906ed089; no voters:
I20260620 18:46:31.734683 23194 leader_election.cc:288] T 00000000000000000000000000000000 P b00294c8658947519fc4c046906ed089 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:46:31.734782 23199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b00294c8658947519fc4c046906ed089 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:46:31.735505 23199 raft_consensus.cc:686] T 00000000000000000000000000000000 P b00294c8658947519fc4c046906ed089 [term 1 LEADER]: Becoming Leader. State: Replica: b00294c8658947519fc4c046906ed089, State: Running, Role: LEADER
I20260620 18:46:31.736074 23199 consensus_queue.cc:237] T 00000000000000000000000000000000 P b00294c8658947519fc4c046906ed089 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00294c8658947519fc4c046906ed089" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 37333 } }
I20260620 18:46:31.736948 23194 sys_catalog.cc:564] T 00000000000000000000000000000000 P b00294c8658947519fc4c046906ed089 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:46:31.739195 23201 sys_catalog.cc:455] T 00000000000000000000000000000000 P b00294c8658947519fc4c046906ed089 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b00294c8658947519fc4c046906ed089. Latest consensus state: current_term: 1 leader_uuid: "b00294c8658947519fc4c046906ed089" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00294c8658947519fc4c046906ed089" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 37333 } } }
I20260620 18:46:31.739530 23201 sys_catalog.cc:458] T 00000000000000000000000000000000 P b00294c8658947519fc4c046906ed089 [sys.catalog]: This master's current role is: LEADER
I20260620 18:46:31.740078 23200 sys_catalog.cc:455] T 00000000000000000000000000000000 P b00294c8658947519fc4c046906ed089 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b00294c8658947519fc4c046906ed089" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00294c8658947519fc4c046906ed089" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 37333 } } }
I20260620 18:46:31.740382 23200 sys_catalog.cc:458] T 00000000000000000000000000000000 P b00294c8658947519fc4c046906ed089 [sys.catalog]: This master's current role is: LEADER
I20260620 18:46:31.740482 23204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:46:31.746353 23204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:46:31.756541 23204 catalog_manager.cc:1347] Generated new cluster ID: abe40692d6a247d38842b10d7de70bb3
I20260620 18:46:31.756688 23204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:46:31.784816 23204 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:46:31.786795 23204 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:46:31.806955 23204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b00294c8658947519fc4c046906ed089: Generated new TSK 0
I20260620 18:46:31.807950 23204 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab.
Entry for principal kudu/127.5.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab.
I20260620 18:46:31.880208 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.129:0
--local_ip_for_outbound_sockets=127.5.225.129
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:37333
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:39001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:46:32.211522 23220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:46:32.211826 23220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:46:32.212004 23220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:46:32.218982 23220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:46:32.219132 23220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:46:32.219317 23220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.129
I20260620 18:46:32.229115 23220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:39001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:37333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.129
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:46:32.230583 23220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:46:32.231843 23220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:46:32.245184 23225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:32.245262 23226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:32.245225 23228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:46:32.246012 23220 server_base.cc:1034] running on GCE node
I20260620 18:46:32.246635 23220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:46:32.247500 23220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:46:32.248697 23220 hybrid_clock.cc:648] HybridClock initialized: now 1781981192248649 us; error 41 us; skew 500 ppm
I20260620 18:46:32.249002 23220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:46:32 dist-test-slave-k6lg krb5kdc[23114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981192, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:46:32.252195 23220 init.cc:377] Logged in from keytab as kudu/127.5.225.129@KRBTEST.COM (short username kudu)
I20260620 18:46:32.253566 23220 webserver.cc:460] Webserver started at http://127.5.225.129:46263/ using document root <none> and password file <none>
I20260620 18:46:32.254312 23220 fs_manager.cc:359] Metadata directory not provided
I20260620 18:46:32.254446 23220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:46:32.254770 23220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:46:32.257628 23220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-0/data/instance:
uuid: "77328f1a40c648b4939e0c2fc2c941be"
format_stamp: "Formatted at 2026-06-20 18:46:32 on dist-test-slave-k6lg"
server_key: "87b8fe2667d14c83d280aecb3d2b2eb9"
server_key_iv: "8d7516cdf8687d153f558ebf428ae8d6"
server_key_version: "encryptionkey@0"
I20260620 18:46:32.258476 23220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance:
uuid: "77328f1a40c648b4939e0c2fc2c941be"
format_stamp: "Formatted at 2026-06-20 18:46:32 on dist-test-slave-k6lg"
server_key: "87b8fe2667d14c83d280aecb3d2b2eb9"
server_key_iv: "8d7516cdf8687d153f558ebf428ae8d6"
server_key_version: "encryptionkey@0"
I20260620 18:46:32.263630 23220 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:46:32.266994 23235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:32.268244 23220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:46:32.268409 23220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-0/wal
uuid: "77328f1a40c648b4939e0c2fc2c941be"
format_stamp: "Formatted at 2026-06-20 18:46:32 on dist-test-slave-k6lg"
server_key: "87b8fe2667d14c83d280aecb3d2b2eb9"
server_key_iv: "8d7516cdf8687d153f558ebf428ae8d6"
server_key_version: "encryptionkey@0"
I20260620 18:46:32.268599 23220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:46:32.298624 23220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:46:32.303184 23220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:46:32.303488 23220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:46:32.304494 23220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:46:32.305891 23220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:46:32.305997 23220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:32.306156 23220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:46:32.306267 23220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 18:46:32.338750 23220 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.129:38675
I20260620 18:46:32.338800 23347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.129:38675 every 8 connection(s)
I20260620 18:46:32.340114 23220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
I20260620 18:46:32.346139 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 23220
I20260620 18:46:32.346446 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance
I20260620 18:46:32.347093 6022 external_mini_cluster.cc:1428] Setting key ad92d40c4dfb66a9f8aa84e117010493
Jun 20 18:46:32 dist-test-slave-k6lg krb5kdc[23114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981192, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:46:32.363840 23350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:32.342757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:58129 (local address 127.5.225.190:37333)
0620 18:46:32.343272 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:32.343277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:32.344258 (+ 981us) server_negotiation.cc:407] Connection header received
0620 18:46:32.346151 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:32.346156 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:32.346275 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:32.346398 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:32.348062 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:32.349138 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:32.352063 (+ 2925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:32.352482 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:32.352583 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:32.356962 (+ 4379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:32.356973 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:32.356981 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:46:32.357033 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:32.360337 (+ 3304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:32.361291 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:32.361295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:32.361297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:32.361425 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:32.361966 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:32.361970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:32.361972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:32.362349 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:32.362485 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:32.363364 (+ 879us) server_negotiation.cc:299] Negotiation successful
0620 18:46:32.363550 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":261,"threads_started":1}
I20260620 18:46:32.365322 23351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:32.343337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:37333 (local address 127.5.225.129:58129)
0620 18:46:32.344081 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:32.344137 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:32.345870 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:32.346668 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:32.346676 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:32.347170 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:46:32.347877 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:32.347905 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:32.349351 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:32.349358 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:32.351881 (+ 2523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:32.351903 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:32.352730 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:32.352740 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:32.353111 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:32.353886 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:46:32.353931 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:46:32.356714 (+ 2783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:46:32.360492 (+ 3778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:46:32.360503 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:46:32.360523 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:46:32.361168 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:46:32.361534 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:46:32.361541 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:46:32.361548 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:46:32.361791 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:46:32.362489 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:46:32.362499 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:46:32.362945 (+ 446us) client_negotiation.cc:769] Sending connection context
0620 18:46:32.363256 (+ 311us) client_negotiation.cc:240] Negotiation successful
0620 18:46:32.363826 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":312,"threads_started":1}
I20260620 18:46:32.367026 23348 heartbeater.cc:347] Connected to a master server at 127.5.225.190:37333
I20260620 18:46:32.367360 23348 heartbeater.cc:464] Registering TS with master...
I20260620 18:46:32.368100 23348 heartbeater.cc:511] Master 127.5.225.190:37333 requested a full tablet report, sending...
I20260620 18:46:32.370434 23159 ts_manager.cc:194] Registered new tserver with Master: 77328f1a40c648b4939e0c2fc2c941be (127.5.225.129:38675)
I20260620 18:46:32.372140 23159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.129@KRBTEST.COM'} at 127.5.225.129:58129
WARNING: no policy specified for kudu/127.5.225.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab.
Entry for principal kudu/127.5.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab.
I20260620 18:46:32.405385 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.130:0
--local_ip_for_outbound_sockets=127.5.225.130
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:37333
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:39001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:46:32.732204 23354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:46:32.732540 23354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:46:32.732713 23354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:46:32.740198 23354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:46:32.740345 23354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:46:32.740533 23354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.130
I20260620 18:46:32.750401 23354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:39001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:37333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.130
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:46:32.751744 23354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:46:32.753089 23354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:46:32.766636 23362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:32.766709 23359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:32.766686 23360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:46:32.768069 23354 server_base.cc:1034] running on GCE node
I20260620 18:46:32.768680 23354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:46:32.769624 23354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:46:32.770850 23354 hybrid_clock.cc:648] HybridClock initialized: now 1781981192770777 us; error 67 us; skew 500 ppm
I20260620 18:46:32.771163 23354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:46:32 dist-test-slave-k6lg krb5kdc[23114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981192, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:46:32.774616 23354 init.cc:377] Logged in from keytab as kudu/127.5.225.130@KRBTEST.COM (short username kudu)
I20260620 18:46:32.776124 23354 webserver.cc:460] Webserver started at http://127.5.225.130:45841/ using document root <none> and password file <none>
I20260620 18:46:32.776811 23354 fs_manager.cc:359] Metadata directory not provided
I20260620 18:46:32.776966 23354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:46:32.777281 23354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:46:32.780290 23354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-1/data/instance:
uuid: "e909a58a087e4f7aabf108a737552c81"
format_stamp: "Formatted at 2026-06-20 18:46:32 on dist-test-slave-k6lg"
server_key: "84f74e98b1ef1601236f2318502ef323"
server_key_iv: "cbd90564be92849204688d6ec0ff11d5"
server_key_version: "encryptionkey@0"
I20260620 18:46:32.781067 23354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance:
uuid: "e909a58a087e4f7aabf108a737552c81"
format_stamp: "Formatted at 2026-06-20 18:46:32 on dist-test-slave-k6lg"
server_key: "84f74e98b1ef1601236f2318502ef323"
server_key_iv: "cbd90564be92849204688d6ec0ff11d5"
server_key_version: "encryptionkey@0"
I20260620 18:46:32.786379 23354 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260620 18:46:32.789884 23369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:32.791266 23354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:46:32.791448 23354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-1/wal
uuid: "e909a58a087e4f7aabf108a737552c81"
format_stamp: "Formatted at 2026-06-20 18:46:32 on dist-test-slave-k6lg"
server_key: "84f74e98b1ef1601236f2318502ef323"
server_key_iv: "cbd90564be92849204688d6ec0ff11d5"
server_key_version: "encryptionkey@0"
I20260620 18:46:32.791658 23354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:46:32.824710 23354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:46:32.829588 23354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:46:32.829907 23354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:46:32.830948 23354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:46:32.832326 23354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:46:32.832430 23354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:32.832581 23354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:46:32.832664 23354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:32.867666 23354 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.130:40387
I20260620 18:46:32.867754 23481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.130:40387 every 8 connection(s)
I20260620 18:46:32.868986 23354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
I20260620 18:46:32.869668 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 23354
I20260620 18:46:32.869896 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance
I20260620 18:46:32.870612 6022 external_mini_cluster.cc:1428] Setting key aedd64b29bc53c2b094509327a04d909
Jun 20 18:46:32 dist-test-slave-k6lg krb5kdc[23114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981192, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:46:32.895494 23484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:32.871874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:47533 (local address 127.5.225.190:37333)
0620 18:46:32.872384 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:32.872390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:32.873718 (+ 1328us) server_negotiation.cc:407] Connection header received
0620 18:46:32.877171 (+ 3453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:32.877175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:32.877245 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:32.877345 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:32.879655 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:32.880395 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:32.883228 (+ 2833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:32.883982 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:32.884165 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:32.887957 (+ 3792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:32.887966 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:32.887974 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:46:32.888021 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:32.891821 (+ 3800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:32.892631 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:32.892640 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:32.892646 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:32.892805 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:32.893286 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:32.893294 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:32.893299 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:32.893696 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:32.893852 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:32.894689 (+ 837us) server_negotiation.cc:299] Negotiation successful
0620 18:46:32.894948 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":266,"threads_started":1}
I20260620 18:46:32.896478 23485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:32.872587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:37333 (local address 127.5.225.130:47533)
0620 18:46:32.873528 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:32.873585 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:32.876791 (+ 3206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:32.877652 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:32.877669 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:32.878569 (+ 900us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:46:32.879463 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:32.879493 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:32.880572 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:32.880582 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:32.883056 (+ 2474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:32.883077 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:32.884215 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:32.884220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:32.884448 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:32.885205 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:46:32.885230 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:46:32.887673 (+ 2443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:46:32.891990 (+ 4317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:46:32.891997 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:46:32.892012 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:46:32.892484 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:46:32.892949 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:46:32.892953 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:46:32.892956 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:46:32.893154 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:46:32.893820 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:46:32.893827 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:46:32.894272 (+ 445us) client_negotiation.cc:769] Sending connection context
0620 18:46:32.894532 (+ 260us) client_negotiation.cc:240] Negotiation successful
0620 18:46:32.895112 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":381,"threads_started":1}
I20260620 18:46:32.898118 23482 heartbeater.cc:347] Connected to a master server at 127.5.225.190:37333
I20260620 18:46:32.898494 23482 heartbeater.cc:464] Registering TS with master...
I20260620 18:46:32.899266 23482 heartbeater.cc:511] Master 127.5.225.190:37333 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.225.131@KRBTEST.COM; defaulting to no policy
I20260620 18:46:32.900944 23159 ts_manager.cc:194] Registered new tserver with Master: e909a58a087e4f7aabf108a737552c81 (127.5.225.130:40387)
I20260620 18:46:32.901966 23159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.130@KRBTEST.COM'} at 127.5.225.130:47533
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab.
Entry for principal kudu/127.5.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab.
I20260620 18:46:32.931098 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.131:0
--local_ip_for_outbound_sockets=127.5.225.131
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:37333
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:39001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:46:33.262590 23488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:46:33.262848 23488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:46:33.262981 23488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:46:33.270351 23488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:46:33.270458 23488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:46:33.270572 23488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.131
I20260620 18:46:33.280539 23488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:39001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:37333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.131
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:46:33.281747 23488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:46:33.283162 23488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:46:33.295938 23493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:33.296152 23494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:33.296638 23496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:46:33.297560 23488 server_base.cc:1034] running on GCE node
I20260620 18:46:33.298105 23488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:46:33.299103 23488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:46:33.300294 23488 hybrid_clock.cc:648] HybridClock initialized: now 1781981193300255 us; error 25 us; skew 500 ppm
I20260620 18:46:33.300642 23488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:46:33 dist-test-slave-k6lg krb5kdc[23114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981193, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:46:33.303869 23488 init.cc:377] Logged in from keytab as kudu/127.5.225.131@KRBTEST.COM (short username kudu)
I20260620 18:46:33.305203 23488 webserver.cc:460] Webserver started at http://127.5.225.131:32889/ using document root <none> and password file <none>
I20260620 18:46:33.305831 23488 fs_manager.cc:359] Metadata directory not provided
I20260620 18:46:33.305929 23488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:46:33.306257 23488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:46:33.308996 23488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-2/data/instance:
uuid: "1b4c54e6829e4261b13eab925c48edb9"
format_stamp: "Formatted at 2026-06-20 18:46:33 on dist-test-slave-k6lg"
server_key: "a5713cd4de8f2b9037ebd66790f50916"
server_key_iv: "faab4ad6a7874dcbdad9f9456a5f01de"
server_key_version: "encryptionkey@0"
I20260620 18:46:33.309707 23488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance:
uuid: "1b4c54e6829e4261b13eab925c48edb9"
format_stamp: "Formatted at 2026-06-20 18:46:33 on dist-test-slave-k6lg"
server_key: "a5713cd4de8f2b9037ebd66790f50916"
server_key_iv: "faab4ad6a7874dcbdad9f9456a5f01de"
server_key_version: "encryptionkey@0"
I20260620 18:46:33.314558 23488 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 18:46:33.317752 23503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:33.319198 23488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:46:33.319379 23488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-2/wal
uuid: "1b4c54e6829e4261b13eab925c48edb9"
format_stamp: "Formatted at 2026-06-20 18:46:33 on dist-test-slave-k6lg"
server_key: "a5713cd4de8f2b9037ebd66790f50916"
server_key_iv: "faab4ad6a7874dcbdad9f9456a5f01de"
server_key_version: "encryptionkey@0"
I20260620 18:46:33.319582 23488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:46:33.351459 23488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:46:33.355924 23488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:46:33.356230 23488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:46:33.357237 23488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:46:33.358673 23488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:46:33.358745 23488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:33.358920 23488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:46:33.359010 23488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:33.375519 23348 heartbeater.cc:503] Master 127.5.225.190:37333 was elected leader, sending a full tablet report...
I20260620 18:46:33.392182 23488 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.131:42491
I20260620 18:46:33.392227 23615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.131:42491 every 8 connection(s)
I20260620 18:46:33.393487 23488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
I20260620 18:46:33.398897 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 23488
I20260620 18:46:33.399075 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance
I20260620 18:46:33.399724 6022 external_mini_cluster.cc:1428] Setting key 8f5b16fef4a501ba1dc1fc4dbadf233c
Jun 20 18:46:33 dist-test-slave-k6lg krb5kdc[23114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981193, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:46:33.416121 23618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:33.396438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:43855 (local address 127.5.225.190:37333)
0620 18:46:33.397012 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:33.397018 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:33.398288 (+ 1270us) server_negotiation.cc:407] Connection header received
0620 18:46:33.401050 (+ 2762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:33.401055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:33.401169 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:33.401286 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:33.403666 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:33.404894 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:33.406519 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:33.406910 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:33.407047 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:33.410379 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:33.410384 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:33.410387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:46:33.410418 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:33.412839 (+ 2421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:33.413526 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:33.413531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:33.413534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:33.413657 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:33.414140 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:33.414143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:33.414145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:33.414520 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:33.414658 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:33.415525 (+ 867us) server_negotiation.cc:299] Negotiation successful
0620 18:46:33.415745 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":290,"threads_started":1}
I20260620 18:46:33.417019 23619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:33.397081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:37333 (local address 127.5.225.131:43855)
0620 18:46:33.398058 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:33.398134 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:33.400712 (+ 2578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:33.401766 (+ 1054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:33.401781 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:33.402594 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:46:33.403439 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:33.403493 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:33.405048 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:33.405054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:33.406371 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:33.406383 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:33.407134 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:33.407143 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:33.407379 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:33.407958 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:46:33.407980 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:46:33.410130 (+ 2150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:46:33.412963 (+ 2833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:46:33.412969 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:46:33.412981 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:46:33.413397 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:46:33.413768 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:46:33.413773 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:46:33.413780 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:46:33.414010 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:46:33.414668 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:46:33.414676 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:46:33.415113 (+ 437us) client_negotiation.cc:769] Sending connection context
0620 18:46:33.415407 (+ 294us) client_negotiation.cc:240] Negotiation successful
0620 18:46:33.415898 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":342,"threads_started":1}
I20260620 18:46:33.418357 23616 heartbeater.cc:347] Connected to a master server at 127.5.225.190:37333
I20260620 18:46:33.418653 23616 heartbeater.cc:464] Registering TS with master...
I20260620 18:46:33.419349 23616 heartbeater.cc:511] Master 127.5.225.190:37333 requested a full tablet report, sending...
I20260620 18:46:33.420662 23159 ts_manager.cc:194] Registered new tserver with Master: 1b4c54e6829e4261b13eab925c48edb9 (127.5.225.131:42491)
I20260620 18:46:33.421458 23159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.131@KRBTEST.COM'} at 127.5.225.131:43855
I20260620 18:46:33.428030 6022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:46:33.446278 23618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:33.432555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40470 (local address 127.5.225.190:37333)
0620 18:46:33.432733 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:33.432737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:33.433131 (+ 394us) server_negotiation.cc:407] Connection header received
0620 18:46:33.433249 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:33.433253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:33.433311 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:33.433390 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:33.434869 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:33.435584 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:33.437114 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:33.437870 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:33.438017 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:33.440715 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:33.440720 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:33.440723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:46:33.440757 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:33.443548 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:33.444367 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:33.444371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:33.444373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:33.444491 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:33.444872 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:33.444876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:33.444877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:33.445205 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:33.445415 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:33.445755 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 18:46:33.445932 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:46:33.450589 23159 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40470:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:46:33.457486 23159 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:46:33.488864 23631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:33.470139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55108 (local address 127.5.225.131:42491)
0620 18:46:33.471320 (+ 1181us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:33.471328 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:33.471359 (+ 31us) server_negotiation.cc:407] Connection header received
0620 18:46:33.471461 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:33.471471 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:33.477982 (+ 6511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:33.478765 (+ 783us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:46:33.480430 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:33.482583 (+ 2153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:33.486150 (+ 3567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:33.487317 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:33.487448 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:33.487681 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:33.488323 (+ 642us) server_negotiation.cc:299] Negotiation successful
0620 18:46:33.488493 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1009,"thread_start_us":829,"threads_started":1}
I20260620 18:46:33.488979 23629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:33.469676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.131:42491 (local address 127.0.0.1:55108)
0620 18:46:33.470631 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:33.470667 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:33.470837 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:33.479176 (+ 8339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:33.479187 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:33.479232 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:46:33.479729 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:33.479747 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:33.482810 (+ 3063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:33.482819 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:33.486017 (+ 3198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:33.486033 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:33.487837 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:33.487845 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:33.488084 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:33.488121 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 18:46:33.488278 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 18:46:33.488500 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"mutex_wait_us":1,"thread_start_us":581,"threads_started":1}
I20260620 18:46:33.491335 23630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:33.469819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.130:40387 (local address 127.0.0.1:49096)
0620 18:46:33.481821 (+ 12002us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:33.481845 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:33.482016 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:33.483174 (+ 1158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:33.483182 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:33.483220 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:46:33.484116 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:33.484137 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:33.486070 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:33.486078 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:33.488286 (+ 2208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:33.488303 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:33.490462 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:33.490479 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:33.490668 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:33.490708 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 18:46:33.490788 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 18:46:33.490930 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11862,"thread_start_us":11731,"threads_started":1}
I20260620 18:46:33.492057 23632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:33.470622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49096 (local address 127.5.225.130:40387)
0620 18:46:33.481169 (+ 10547us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:33.481181 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:33.482188 (+ 1007us) server_negotiation.cc:407] Connection header received
0620 18:46:33.482331 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:33.482342 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:33.482636 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:33.482866 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:46:33.484361 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:33.485932 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:33.488467 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:33.489937 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:33.490049 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:33.490265 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:33.491562 (+ 1297us) server_negotiation.cc:299] Negotiation successful
0620 18:46:33.491732 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10324,"thread_start_us":10550,"threads_started":1}
I20260620 18:46:33.493408 23633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:33.471755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:38675 (local address 127.0.0.1:37710)
0620 18:46:33.484635 (+ 12880us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:33.484652 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:33.484768 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:33.485402 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:33.485406 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:33.485431 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:46:33.485803 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:33.485812 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:33.487811 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:33.487819 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:33.490575 (+ 2756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:33.490587 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:33.492612 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:33.492619 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:33.492789 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:33.492822 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 18:46:33.492940 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 18:46:33.493061 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12757,"thread_start_us":12572,"threads_started":1}
I20260620 18:46:33.493480 23634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:33.472028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37710 (local address 127.5.225.129:38675)
0620 18:46:33.484139 (+ 12111us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:33.484146 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:33.484898 (+ 752us) server_negotiation.cc:407] Connection header received
0620 18:46:33.484993 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:33.484998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:33.485235 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:33.485416 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:46:33.485960 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:33.487631 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:33.490760 (+ 3129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:33.492402 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:33.492563 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:33.492804 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:33.492993 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 18:46:33.493144 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11949,"thread_start_us":11763,"threads_started":1}
I20260620 18:46:33.493131 23551 tablet_service.cc:1467] Processing CreateTablet for tablet 64877d291d2e46f2a370509a4fb93a87 (DEFAULT_TABLE table=test-table [id=b73e14d5dd5f4be5981d4e7c78535a33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:46:33.494657 23551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64877d291d2e46f2a370509a4fb93a87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:46:33.496975 23283 tablet_service.cc:1467] Processing CreateTablet for tablet 64877d291d2e46f2a370509a4fb93a87 (DEFAULT_TABLE table=test-table [id=b73e14d5dd5f4be5981d4e7c78535a33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:46:33.496975 23417 tablet_service.cc:1467] Processing CreateTablet for tablet 64877d291d2e46f2a370509a4fb93a87 (DEFAULT_TABLE table=test-table [id=b73e14d5dd5f4be5981d4e7c78535a33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:46:33.498355 23283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64877d291d2e46f2a370509a4fb93a87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:46:33.498361 23417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64877d291d2e46f2a370509a4fb93a87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:46:33.507424 23635 tablet_bootstrap.cc:492] T 64877d291d2e46f2a370509a4fb93a87 P 1b4c54e6829e4261b13eab925c48edb9: Bootstrap starting.
I20260620 18:46:33.510457 23635 tablet_bootstrap.cc:654] T 64877d291d2e46f2a370509a4fb93a87 P 1b4c54e6829e4261b13eab925c48edb9: Neither blocks nor log segments found. Creating new log.
I20260620 18:46:33.511461 23635 log.cc:824] T 64877d291d2e46f2a370509a4fb93a87 P 1b4c54e6829e4261b13eab925c48edb9: Log is configured to *not* fsync() on all Append() calls
I20260620 18:46:33.513599 23637 tablet_bootstrap.cc:492] T 64877d291d2e46f2a370509a4fb93a87 P 77328f1a40c648b4939e0c2fc2c941be: Bootstrap starting.
I20260620 18:46:33.514734 23635 tablet_bootstrap.cc:492] T 64877d291d2e46f2a370509a4fb93a87 P 1b4c54e6829e4261b13eab925c48edb9: No bootstrap required, opened a new log
I20260620 18:46:33.514837 23638 tablet_bootstrap.cc:492] T 64877d291d2e46f2a370509a4fb93a87 P e909a58a087e4f7aabf108a737552c81: Bootstrap starting.
I20260620 18:46:33.515018 23635 ts_tablet_manager.cc:1397] T 64877d291d2e46f2a370509a4fb93a87 P 1b4c54e6829e4261b13eab925c48edb9: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 18:46:33.517326 23637 tablet_bootstrap.cc:654] T 64877d291d2e46f2a370509a4fb93a87 P 77328f1a40c648b4939e0c2fc2c941be: Neither blocks nor log segments found. Creating new log.
I20260620 18:46:33.517565 23638 tablet_bootstrap.cc:654] T 64877d291d2e46f2a370509a4fb93a87 P e909a58a087e4f7aabf108a737552c81: Neither blocks nor log segments found. Creating new log.
I20260620 18:46:33.518594 23638 log.cc:824] T 64877d291d2e46f2a370509a4fb93a87 P e909a58a087e4f7aabf108a737552c81: Log is configured to *not* fsync() on all Append() calls
I20260620 18:46:33.518592 23637 log.cc:824] T 64877d291d2e46f2a370509a4fb93a87 P 77328f1a40c648b4939e0c2fc2c941be: Log is configured to *not* fsync() on all Append() calls
I20260620 18:46:33.521401 23638 tablet_bootstrap.cc:492] T 64877d291d2e46f2a370509a4fb93a87 P e909a58a087e4f7aabf108a737552c81: No bootstrap required, opened a new log
I20260620 18:46:33.521675 23638 ts_tablet_manager.cc:1397] T 64877d291d2e46f2a370509a4fb93a87 P e909a58a087e4f7aabf108a737552c81: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 18:46:33.522313 23637 tablet_bootstrap.cc:492] T 64877d291d2e46f2a370509a4fb93a87 P 77328f1a40c648b4939e0c2fc2c941be: No bootstrap required, opened a new log
I20260620 18:46:33.522569 23637 ts_tablet_manager.cc:1397] T 64877d291d2e46f2a370509a4fb93a87 P 77328f1a40c648b4939e0c2fc2c941be: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20260620 18:46:33.525758 23635 raft_consensus.cc:348] T 64877d291d2e46f2a370509a4fb93a87 P 1b4c54e6829e4261b13eab925c48edb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4c54e6829e4261b13eab925c48edb9" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 42491 } } peers { permanent_uuid: "e909a58a087e4f7aabf108a737552c81" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40387 } } peers { permanent_uuid: "77328f1a40c648b4939e0c2fc2c941be" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 38675 } }
I20260620 18:46:33.526232 23635 raft_consensus.cc:374] T 64877d291d2e46f2a370509a4fb93a87 P 1b4c54e6829e4261b13eab925c48edb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:46:33.526340 23635 raft_consensus.cc:729] T 64877d291d2e46f2a370509a4fb93a87 P 1b4c54e6829e4261b13eab925c48edb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b4c54e6829e4261b13eab925c48edb9, State: Initialized, Role: FOLLOWER
I20260620 18:46:33.526988 23635 consensus_queue.cc:260] T 64877d291d2e46f2a370509a4fb93a87 P 1b4c54e6829e4261b13eab925c48edb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4c54e6829e4261b13eab925c48edb9" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 42491 } } peers { permanent_uuid: "e909a58a087e4f7aabf108a737552c81" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40387 } } peers { permanent_uuid: "77328f1a40c648b4939e0c2fc2c941be" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 38675 } }
I20260620 18:46:33.528170 23616 heartbeater.cc:503] Master 127.5.225.190:37333 was elected leader, sending a full tablet report...
I20260620 18:46:33.529017 23635 ts_tablet_manager.cc:1428] T 64877d291d2e46f2a370509a4fb93a87 P 1b4c54e6829e4261b13eab925c48edb9: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260620 18:46:33.533205 23638 raft_consensus.cc:348] T 64877d291d2e46f2a370509a4fb93a87 P e909a58a087e4f7aabf108a737552c81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4c54e6829e4261b13eab925c48edb9" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 42491 } } peers { permanent_uuid: "e909a58a087e4f7aabf108a737552c81" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40387 } } peers { permanent_uuid: "77328f1a40c648b4939e0c2fc2c941be" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 38675 } }
I20260620 18:46:33.533677 23638 raft_consensus.cc:374] T 64877d291d2e46f2a370509a4fb93a87 P e909a58a087e4f7aabf108a737552c81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:46:33.533792 23638 raft_consensus.cc:729] T 64877d291d2e46f2a370509a4fb93a87 P e909a58a087e4f7aabf108a737552c81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e909a58a087e4f7aabf108a737552c81, State: Initialized, Role: FOLLOWER
I20260620 18:46:33.533205 23637 raft_consensus.cc:348] T 64877d291d2e46f2a370509a4fb93a87 P 77328f1a40c648b4939e0c2fc2c941be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4c54e6829e4261b13eab925c48edb9" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 42491 } } peers { permanent_uuid: "e909a58a087e4f7aabf108a737552c81" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40387 } } peers { permanent_uuid: "77328f1a40c648b4939e0c2fc2c941be" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 38675 } }
I20260620 18:46:33.534117 23637 raft_consensus.cc:374] T 64877d291d2e46f2a370509a4fb93a87 P 77328f1a40c648b4939e0c2fc2c941be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:46:33.534233 23637 raft_consensus.cc:729] T 64877d291d2e46f2a370509a4fb93a87 P 77328f1a40c648b4939e0c2fc2c941be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77328f1a40c648b4939e0c2fc2c941be, State: Initialized, Role: FOLLOWER
I20260620 18:46:33.534556 23638 consensus_queue.cc:260] T 64877d291d2e46f2a370509a4fb93a87 P e909a58a087e4f7aabf108a737552c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4c54e6829e4261b13eab925c48edb9" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 42491 } } peers { permanent_uuid: "e909a58a087e4f7aabf108a737552c81" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40387 } } peers { permanent_uuid: "77328f1a40c648b4939e0c2fc2c941be" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 38675 } }
I20260620 18:46:33.534795 23637 consensus_queue.cc:260] T 64877d291d2e46f2a370509a4fb93a87 P 77328f1a40c648b4939e0c2fc2c941be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4c54e6829e4261b13eab925c48edb9" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 42491 } } peers { permanent_uuid: "e909a58a087e4f7aabf108a737552c81" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40387 } } peers { permanent_uuid: "77328f1a40c648b4939e0c2fc2c941be" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 38675 } }
I20260620 18:46:33.535758 23482 heartbeater.cc:503] Master 127.5.225.190:37333 was elected leader, sending a full tablet report...
I20260620 18:46:33.536482 23638 ts_tablet_manager.cc:1428] T 64877d291d2e46f2a370509a4fb93a87 P e909a58a087e4f7aabf108a737552c81: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
I20260620 18:46:33.536484 23637 ts_tablet_manager.cc:1428] T 64877d291d2e46f2a370509a4fb93a87 P 77328f1a40c648b4939e0c2fc2c941be: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
W20260620 18:46:33.592576 23349 tablet.cc:2369] T 64877d291d2e46f2a370509a4fb93a87 P 77328f1a40c648b4939e0c2fc2c941be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:46:33.596019 23641 raft_consensus.cc:482] T 64877d291d2e46f2a370509a4fb93a87 P 1b4c54e6829e4261b13eab925c48edb9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:46:33.596342 23641 raft_consensus.cc:504] T 64877d291d2e46f2a370509a4fb93a87 P 1b4c54e6829e4261b13eab925c48edb9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4c54e6829e4261b13eab925c48edb9" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 42491 } } peers { permanent_uuid: "e909a58a087e4f7aabf108a737552c81" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40387 } } peers { permanent_uuid: "77328f1a40c648b4939e0c2fc2c941be" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 38675 } }
I20260620 18:46:33.597960 23641 leader_election.cc:288] T 64877d291d2e46f2a370509a4fb93a87 P 1b4c54e6829e4261b13eab925c48edb9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e909a58a087e4f7aabf108a737552c81 (127.5.225.130:40387), 77328f1a40c648b4939e0c2fc2c941be (127.5.225.129:38675)
I20260620 18:46:33.604423 23619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:33.598291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:38675 (local address 127.5.225.131:43001)
0620 18:46:33.598604 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:33.598628 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:33.598813 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:33.599254 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:33.599260 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:33.599291 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:46:33.599496 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:33.599512 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:33.600801 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:33.600807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:33.602331 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:33.602343 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:33.603645 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:33.603651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:33.603842 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:33.603877 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 18:46:33.603981 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 18:46:33.604160 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150}
I20260620 18:46:33.604511 23634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:33.598412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:43001 (local address 127.5.225.129:38675)
0620 18:46:33.598630 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:33.598635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:33.598732 (+ 97us) server_negotiation.cc:407] Connection header received
0620 18:46:33.598997 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:33.599002 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:33.599095 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:33.599209 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:46:33.599660 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:33.600665 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:33.602491 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:33.603463 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:33.603612 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:33.603664 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:33.604080 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 18:46:33.604227 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 18:46:33.604939 23632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:33.598424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:60935 (local address 127.5.225.130:40387)
0620 18:46:33.598629 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:33.598635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:33.599222 (+ 587us) server_negotiation.cc:407] Connection header received
0620 18:46:33.599350 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:33.599356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:33.599435 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:33.599536 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:46:33.600254 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:33.601079 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:33.603024 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:33.603988 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:33.604121 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:33.604185 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:33.604610 (+ 425us) server_negotiation.cc:299] Negotiation successful
0620 18:46:33.604749 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:46:33.605602 23644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:33.598334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.130:40387 (local address 127.5.225.131:60935)
0620 18:46:33.599103 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:33.599124 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:33.599237 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:33.599636 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:33.599641 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:33.599668 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:46:33.600090 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:33.600104 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:33.601230 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:33.601235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:33.602868 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:33.602884 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:33.604189 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:33.604195 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:33.604368 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:33.604443 (+ 75us) client_negotiation.cc:769] Sending connection context
0620 18:46:33.605124 (+ 681us) client_negotiation.cc:240] Negotiation successful
0620 18:46:33.605270 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":479,"threads_started":1}
I20260620 18:46:33.605645 23303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64877d291d2e46f2a370509a4fb93a87" candidate_uuid: "1b4c54e6829e4261b13eab925c48edb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77328f1a40c648b4939e0c2fc2c941be" is_pre_election: true
I20260620 18:46:33.606274 23303 raft_consensus.cc:2454] T 64877d291d2e46f2a370509a4fb93a87 P 77328f1a40c648b4939e0c2fc2c941be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b4c54e6829e4261b13eab925c48edb9 in term 0.
I20260620 18:46:33.606739 23437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64877d291d2e46f2a370509a4fb93a87" candidate_uuid: "1b4c54e6829e4261b13eab925c48edb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e909a58a087e4f7aabf108a737552c81" is_pre_election: true
I20260620 18:46:33.606995 23507 leader_election.cc:302] T 64877d291d2e46f2a370509a4fb93a87 P 1b4c54e6829e4261b13eab925c48edb9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b4c54e6829e4261b13eab925c48edb9, 77328f1a40c648b4939e0c2fc2c941be; no voters:
I20260620 18:46:33.607302 23437 raft_consensus.cc:2454] T 64877d291d2e46f2a370509a4fb93a87 P e909a58a087e4f7aabf108a737552c81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b4c54e6829e4261b13eab925c48edb9 in term 0.
I20260620 18:46:33.607388 23641 raft_consensus.cc:2781] T 64877d291d2e46f2a370509a4fb93a87 P 1b4c54e6829e4261b13eab925c48edb9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:46:33.607543 23641 raft_consensus.cc:482] T 64877d291d2e46f2a370509a4fb93a87 P 1b4c54e6829e4261b13eab925c48edb9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:46:33.607631 23641 raft_consensus.cc:3037] T 64877d291d2e46f2a370509a4fb93a87 P 1b4c54e6829e4261b13eab925c48edb9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:46:33.609956 23641 raft_consensus.cc:504] T 64877d291d2e46f2a370509a4fb93a87 P 1b4c54e6829e4261b13eab925c48edb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4c54e6829e4261b13eab925c48edb9" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 42491 } } peers { permanent_uuid: "e909a58a087e4f7aabf108a737552c81" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40387 } } peers { permanent_uuid: "77328f1a40c648b4939e0c2fc2c941be" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 38675 } }
I20260620 18:46:33.610596 23641 leader_election.cc:288] T 64877d291d2e46f2a370509a4fb93a87 P 1b4c54e6829e4261b13eab925c48edb9 [CANDIDATE]: Term 1 election: Requested vote from peers e909a58a087e4f7aabf108a737552c81 (127.5.225.130:40387), 77328f1a40c648b4939e0c2fc2c941be (127.5.225.129:38675)
I20260620 18:46:33.610991 23437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64877d291d2e46f2a370509a4fb93a87" candidate_uuid: "1b4c54e6829e4261b13eab925c48edb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e909a58a087e4f7aabf108a737552c81"
I20260620 18:46:33.611191 23437 raft_consensus.cc:3037] T 64877d291d2e46f2a370509a4fb93a87 P e909a58a087e4f7aabf108a737552c81 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:46:33.611187 23303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64877d291d2e46f2a370509a4fb93a87" candidate_uuid: "1b4c54e6829e4261b13eab925c48edb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77328f1a40c648b4939e0c2fc2c941be"
I20260620 18:46:33.611400 23303 raft_consensus.cc:3037] T 64877d291d2e46f2a370509a4fb93a87 P 77328f1a40c648b4939e0c2fc2c941be [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:46:33.613426 23437 raft_consensus.cc:2454] T 64877d291d2e46f2a370509a4fb93a87 P e909a58a087e4f7aabf108a737552c81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b4c54e6829e4261b13eab925c48edb9 in term 1.
I20260620 18:46:33.613638 23303 raft_consensus.cc:2454] T 64877d291d2e46f2a370509a4fb93a87 P 77328f1a40c648b4939e0c2fc2c941be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b4c54e6829e4261b13eab925c48edb9 in term 1.
I20260620 18:46:33.613911 23506 leader_election.cc:302] T 64877d291d2e46f2a370509a4fb93a87 P 1b4c54e6829e4261b13eab925c48edb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b4c54e6829e4261b13eab925c48edb9, e909a58a087e4f7aabf108a737552c81; no voters:
I20260620 18:46:33.614260 23641 raft_consensus.cc:2781] T 64877d291d2e46f2a370509a4fb93a87 P 1b4c54e6829e4261b13eab925c48edb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:46:33.614650 23641 raft_consensus.cc:686] T 64877d291d2e46f2a370509a4fb93a87 P 1b4c54e6829e4261b13eab925c48edb9 [term 1 LEADER]: Becoming Leader. State: Replica: 1b4c54e6829e4261b13eab925c48edb9, State: Running, Role: LEADER
I20260620 18:46:33.615185 23641 consensus_queue.cc:237] T 64877d291d2e46f2a370509a4fb93a87 P 1b4c54e6829e4261b13eab925c48edb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4c54e6829e4261b13eab925c48edb9" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 42491 } } peers { permanent_uuid: "e909a58a087e4f7aabf108a737552c81" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40387 } } peers { permanent_uuid: "77328f1a40c648b4939e0c2fc2c941be" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 38675 } }
I20260620 18:46:33.619837 23158 catalog_manager.cc:5495] T 64877d291d2e46f2a370509a4fb93a87 P 1b4c54e6829e4261b13eab925c48edb9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b4c54e6829e4261b13eab925c48edb9 (127.5.225.131). New cstate: current_term: 1 leader_uuid: "1b4c54e6829e4261b13eab925c48edb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4c54e6829e4261b13eab925c48edb9" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 42491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e909a58a087e4f7aabf108a737552c81" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77328f1a40c648b4939e0c2fc2c941be" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 38675 } health_report { overall_health: UNKNOWN } } }
W20260620 18:46:33.621263 23483 tablet.cc:2369] T 64877d291d2e46f2a370509a4fb93a87 P e909a58a087e4f7aabf108a737552c81: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:46:33.645259 23617 tablet.cc:2369] T 64877d291d2e46f2a370509a4fb93a87 P 1b4c54e6829e4261b13eab925c48edb9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:46:33.649730 23631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:33.644113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55120 (local address 127.5.225.131:42491)
0620 18:46:33.644356 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:33.644362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:33.644382 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:46:33.644628 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:33.644632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:33.644710 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:33.644880 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:46:33.645269 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:33.646132 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:33.647598 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:33.648010 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:33.648144 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:33.648631 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:46:33.649150 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:46:33.649319 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:33.649371 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 18:46:33.649454 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:46:33.657791 23303 raft_consensus.cc:1261] T 64877d291d2e46f2a370509a4fb93a87 P 77328f1a40c648b4939e0c2fc2c941be [term 1 FOLLOWER]: Refusing update from remote peer 1b4c54e6829e4261b13eab925c48edb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:46:33.657801 23437 raft_consensus.cc:1261] T 64877d291d2e46f2a370509a4fb93a87 P e909a58a087e4f7aabf108a737552c81 [term 1 FOLLOWER]: Refusing update from remote peer 1b4c54e6829e4261b13eab925c48edb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:46:33.658944 23645 consensus_queue.cc:1035] T 64877d291d2e46f2a370509a4fb93a87 P 1b4c54e6829e4261b13eab925c48edb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77328f1a40c648b4939e0c2fc2c941be" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 38675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:46:33.659348 23641 consensus_queue.cc:1035] T 64877d291d2e46f2a370509a4fb93a87 P 1b4c54e6829e4261b13eab925c48edb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e909a58a087e4f7aabf108a737552c81" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:46:33.684304 23159 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:46:33.684585 23159 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:46:33.688007 23159 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b00294c8658947519fc4c046906ed089: Sending DeleteTablet for 3 replicas of tablet 64877d291d2e46f2a370509a4fb93a87
I20260620 18:46:33.689216 23551 tablet_service.cc:1514] Processing DeleteTablet for tablet 64877d291d2e46f2a370509a4fb93a87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:46:33 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:55108
I20260620 18:46:33.689231 23283 tablet_service.cc:1514] Processing DeleteTablet for tablet 64877d291d2e46f2a370509a4fb93a87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:46:33 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:37710
I20260620 18:46:33.689278 23417 tablet_service.cc:1514] Processing DeleteTablet for tablet 64877d291d2e46f2a370509a4fb93a87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:46:33 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:49096
I20260620 18:46:33.691959 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 23220
I20260620 18:46:33.706969 23660 tablet_replica.cc:330] T 64877d291d2e46f2a370509a4fb93a87 P 77328f1a40c648b4939e0c2fc2c941be: stopping tablet replica
I20260620 18:46:33.707645 23660 raft_consensus.cc:2229] T 64877d291d2e46f2a370509a4fb93a87 P 77328f1a40c648b4939e0c2fc2c941be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:46:33.708325 23660 raft_consensus.cc:2258] T 64877d291d2e46f2a370509a4fb93a87 P 77328f1a40c648b4939e0c2fc2c941be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:46:33.709372 23658 tablet_replica.cc:330] T 64877d291d2e46f2a370509a4fb93a87 P 1b4c54e6829e4261b13eab925c48edb9: stopping tablet replica
I20260620 18:46:33.709939 23658 raft_consensus.cc:2229] T 64877d291d2e46f2a370509a4fb93a87 P 1b4c54e6829e4261b13eab925c48edb9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:46:33.712028 23660 ts_tablet_manager.cc:1905] T 64877d291d2e46f2a370509a4fb93a87 P 77328f1a40c648b4939e0c2fc2c941be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:46:33.714764 23659 tablet_replica.cc:330] T 64877d291d2e46f2a370509a4fb93a87 P e909a58a087e4f7aabf108a737552c81: stopping tablet replica
I20260620 18:46:33.715152 23658 raft_consensus.cc:2258] T 64877d291d2e46f2a370509a4fb93a87 P 1b4c54e6829e4261b13eab925c48edb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:46:33.715210 23659 raft_consensus.cc:2229] T 64877d291d2e46f2a370509a4fb93a87 P e909a58a087e4f7aabf108a737552c81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:46:33.715754 23659 raft_consensus.cc:2258] T 64877d291d2e46f2a370509a4fb93a87 P e909a58a087e4f7aabf108a737552c81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:46:33.718025 23658 ts_tablet_manager.cc:1905] T 64877d291d2e46f2a370509a4fb93a87 P 1b4c54e6829e4261b13eab925c48edb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:46:33.722020 23660 ts_tablet_manager.cc:1918] T 64877d291d2e46f2a370509a4fb93a87 P 77328f1a40c648b4939e0c2fc2c941be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:46:33.723839 23659 ts_tablet_manager.cc:1905] T 64877d291d2e46f2a370509a4fb93a87 P e909a58a087e4f7aabf108a737552c81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:46:33.725747 23658 ts_tablet_manager.cc:1918] T 64877d291d2e46f2a370509a4fb93a87 P 1b4c54e6829e4261b13eab925c48edb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:46:33.725932 23658 log.cc:1196] T 64877d291d2e46f2a370509a4fb93a87 P 1b4c54e6829e4261b13eab925c48edb9: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-2/wal/wals/64877d291d2e46f2a370509a4fb93a87
I20260620 18:46:33.726531 23658 ts_tablet_manager.cc:1939] T 64877d291d2e46f2a370509a4fb93a87 P 1b4c54e6829e4261b13eab925c48edb9: Deleting consensus metadata
I20260620 18:46:33.728484 23144 catalog_manager.cc:4841] TS 1b4c54e6829e4261b13eab925c48edb9 (127.5.225.131:42491): tablet 64877d291d2e46f2a370509a4fb93a87 (table test-table [id=b73e14d5dd5f4be5981d4e7c78535a33]) successfully deleted
I20260620 18:46:33.729095 23660 log.cc:1196] T 64877d291d2e46f2a370509a4fb93a87 P 77328f1a40c648b4939e0c2fc2c941be: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-0/wal/wals/64877d291d2e46f2a370509a4fb93a87
I20260620 18:46:33.729696 23660 ts_tablet_manager.cc:1939] T 64877d291d2e46f2a370509a4fb93a87 P 77328f1a40c648b4939e0c2fc2c941be: Deleting consensus metadata
Jun 20 18:46:33 dist-test-slave-k6lg krb5kdc[23114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.129@KRBTEST.COM
I20260620 18:46:33.731647 23659 ts_tablet_manager.cc:1918] T 64877d291d2e46f2a370509a4fb93a87 P e909a58a087e4f7aabf108a737552c81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:46:33.731647 23146 catalog_manager.cc:4841] TS 77328f1a40c648b4939e0c2fc2c941be (127.5.225.129:38675): tablet 64877d291d2e46f2a370509a4fb93a87 (table test-table [id=b73e14d5dd5f4be5981d4e7c78535a33]) successfully deleted
I20260620 18:46:33.731828 23659 log.cc:1196] T 64877d291d2e46f2a370509a4fb93a87 P e909a58a087e4f7aabf108a737552c81: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980894474392-6022-0/minicluster-data/ts-1/wal/wals/64877d291d2e46f2a370509a4fb93a87
I20260620 18:46:33.732367 23659 ts_tablet_manager.cc:1939] T 64877d291d2e46f2a370509a4fb93a87 P e909a58a087e4f7aabf108a737552c81: Deleting consensus metadata
I20260620 18:46:33.734526 23145 catalog_manager.cc:4841] TS e909a58a087e4f7aabf108a737552c81 (127.5.225.130:40387): tablet 64877d291d2e46f2a370509a4fb93a87 (table test-table [id=b73e14d5dd5f4be5981d4e7c78535a33]) successfully deleted
I20260620 18:46:33.740103 23634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:33.698543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37716 (local address 127.5.225.129:38675)
0620 18:46:33.698731 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:33.698737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:33.698755 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:46:33.698817 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:33.698822 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:33.698910 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:33.699028 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:33.722345 (+ 23317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:33.723528 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:33.725760 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:33.726424 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:33.726534 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:33.732615 (+ 6081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:33.732631 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:33.732650 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:46:33.732699 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:33.736851 (+ 4152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:33.737750 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:33.737761 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:33.737764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:33.737931 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:33.738414 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:33.738419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:33.738423 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:33.738924 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:33.739243 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:33.739464 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 18:46:33.739696 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 18:46:33.740839 23343 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:46:33.870128 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 23220
I20260620 18:46:33.894537 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 23354
Jun 20 18:46:33 dist-test-slave-k6lg krb5kdc[23114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.130@KRBTEST.COM
I20260620 18:46:33.912760 23632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:33.895380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49112 (local address 127.5.225.130:40387)
0620 18:46:33.895586 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:33.895592 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:33.895708 (+ 116us) server_negotiation.cc:407] Connection header received
0620 18:46:33.895875 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:33.895880 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:33.895961 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:33.896078 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:33.897929 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:33.899105 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:33.900954 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:33.901606 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:33.901757 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:33.906854 (+ 5097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:33.906865 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:33.906880 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:46:33.906917 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:33.909779 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:33.910672 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:33.910686 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:33.910689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:33.910812 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:33.911192 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:33.911199 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:33.911202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:33.911565 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:33.911877 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:33.912190 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 18:46:33.912410 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:46:33.913400 23477 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:46:34.004103 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 23354
I20260620 18:46:34.021015 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 23488
Jun 20 18:46:34 dist-test-slave-k6lg krb5kdc[23114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.131@KRBTEST.COM
I20260620 18:46:34.040146 23631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:34.021812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55132 (local address 127.5.225.131:42491)
0620 18:46:34.022033 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:34.022041 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:34.022070 (+ 29us) server_negotiation.cc:407] Connection header received
0620 18:46:34.022149 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:34.022155 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:34.022283 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:34.022399 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:34.023885 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:34.025144 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:34.026806 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:34.027525 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:34.027680 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:34.032742 (+ 5062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:34.032759 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:34.032786 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:46:34.032844 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:34.036913 (+ 4069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:34.037750 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:34.037765 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:34.037769 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:34.037962 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:34.038487 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:34.038494 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:34.038497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:34.038974 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:34.039310 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:34.039490 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 18:46:34.039725 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:46:34.040995 23611 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:46:34.165634 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 23488
I20260620 18:46:34.181874 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 23127
I20260620 18:46:34.183203 23189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:46:34.294080 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 23127
2026-06-20T18:46:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4376 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T18:46:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:46:34Z Disabled control of system clock
I20260620 18:46:34.369376 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:45753
--webserver_interface=127.5.225.190
--webserver_port=0
--builtin_ntp_servers=127.5.225.148:43471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:45753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:46:34.697541 23669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:46:34.697814 23669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:46:34.697952 23669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:46:34.705245 23669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:46:34.705348 23669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:46:34.705420 23669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:46:34.705513 23669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:46:34.705603 23669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:46:34.714888 23669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:43471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:45753
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:45753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:46:34.716226 23669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:46:34.718042 23669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:46:34.729776 23675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:34.729808 23677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:34.729746 23674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:46:34.730682 23669 server_base.cc:1034] running on GCE node
I20260620 18:46:34.731498 23669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:46:34.732970 23669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:46:34.734266 23669 hybrid_clock.cc:648] HybridClock initialized: now 1781981194734213 us; error 69 us; skew 500 ppm
I20260620 18:46:34.734639 23669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:46:34.737025 23669 webserver.cc:460] Webserver started at http://127.5.225.190:45803/ using document root <none> and password file <none>
I20260620 18:46:34.737722 23669 fs_manager.cc:359] Metadata directory not provided
I20260620 18:46:34.737856 23669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:46:34.738235 23669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:46:34.741061 23669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "fc7f3bb123c348d6837ab2767cdb349a"
format_stamp: "Formatted at 2026-06-20 18:46:34 on dist-test-slave-k6lg"
server_key: "ab48b5495af1a0b9b28b46c053e4128e"
server_key_iv: "f0d048e74a39129dd354868c21e7f51a"
server_key_version: "encryptionkey@0"
I20260620 18:46:34.741853 23669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "fc7f3bb123c348d6837ab2767cdb349a"
format_stamp: "Formatted at 2026-06-20 18:46:34 on dist-test-slave-k6lg"
server_key: "ab48b5495af1a0b9b28b46c053e4128e"
server_key_iv: "f0d048e74a39129dd354868c21e7f51a"
server_key_version: "encryptionkey@0"
I20260620 18:46:34.746903 23669 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260620 18:46:34.750447 23683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:34.751825 23669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:46:34.752019 23669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "fc7f3bb123c348d6837ab2767cdb349a"
format_stamp: "Formatted at 2026-06-20 18:46:34 on dist-test-slave-k6lg"
server_key: "ab48b5495af1a0b9b28b46c053e4128e"
server_key_iv: "f0d048e74a39129dd354868c21e7f51a"
server_key_version: "encryptionkey@0"
I20260620 18:46:34.752241 23669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:46:34.769677 23669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:46:34.770704 23669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:46:34.771008 23669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:46:34.790241 23669 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:45753
I20260620 18:46:34.790262 23734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:45753 every 8 connection(s)
I20260620 18:46:34.791605 23669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:46:34.795337 23735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:46:34.801420 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 23669
I20260620 18:46:34.801607 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:46:34.802042 6022 external_mini_cluster.cc:1428] Setting key 81629f6370db8a9398a16cea79ce38a4
I20260620 18:46:34.807093 23735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc7f3bb123c348d6837ab2767cdb349a: Bootstrap starting.
I20260620 18:46:34.811105 23735 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc7f3bb123c348d6837ab2767cdb349a: Neither blocks nor log segments found. Creating new log.
I20260620 18:46:34.812132 23735 log.cc:824] T 00000000000000000000000000000000 P fc7f3bb123c348d6837ab2767cdb349a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:46:34.815292 23735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc7f3bb123c348d6837ab2767cdb349a: No bootstrap required, opened a new log
I20260620 18:46:34.817502 23738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:34.804132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43488 (local address 127.5.225.190:45753)
0620 18:46:34.805207 (+ 1075us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:34.805222 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:34.805269 (+ 47us) server_negotiation.cc:407] Connection header received
0620 18:46:34.807651 (+ 2382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:34.807677 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:34.808257 (+ 580us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:34.808686 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:34.809278 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:34.811269 (+ 1991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:34.812669 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:34.813446 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:34.813622 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:34.814740 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:34.814750 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:34.814768 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:46:34.815494 (+ 726us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:34.815597 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:46:34.815627 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:34.815755 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:34.815966 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 18:46:34.816290 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":338,"threads_started":1}
I20260620 18:46:34.823716 23735 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc7f3bb123c348d6837ab2767cdb349a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc7f3bb123c348d6837ab2767cdb349a" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 45753 } }
I20260620 18:46:34.824030 23735 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc7f3bb123c348d6837ab2767cdb349a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:46:34.824100 23735 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc7f3bb123c348d6837ab2767cdb349a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc7f3bb123c348d6837ab2767cdb349a, State: Initialized, Role: FOLLOWER
I20260620 18:46:34.824623 23735 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc7f3bb123c348d6837ab2767cdb349a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc7f3bb123c348d6837ab2767cdb349a" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 45753 } }
I20260620 18:46:34.824864 23735 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc7f3bb123c348d6837ab2767cdb349a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:46:34.825026 23735 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc7f3bb123c348d6837ab2767cdb349a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:46:34.825265 23735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc7f3bb123c348d6837ab2767cdb349a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:46:34.827584 23735 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc7f3bb123c348d6837ab2767cdb349a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc7f3bb123c348d6837ab2767cdb349a" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 45753 } }
I20260620 18:46:34.828109 23735 leader_election.cc:302] T 00000000000000000000000000000000 P fc7f3bb123c348d6837ab2767cdb349a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc7f3bb123c348d6837ab2767cdb349a; no voters:
I20260620 18:46:34.828672 23735 leader_election.cc:288] T 00000000000000000000000000000000 P fc7f3bb123c348d6837ab2767cdb349a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:46:34.828830 23740 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc7f3bb123c348d6837ab2767cdb349a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:46:34.829582 23740 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc7f3bb123c348d6837ab2767cdb349a [term 1 LEADER]: Becoming Leader. State: Replica: fc7f3bb123c348d6837ab2767cdb349a, State: Running, Role: LEADER
I20260620 18:46:34.830091 23740 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc7f3bb123c348d6837ab2767cdb349a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc7f3bb123c348d6837ab2767cdb349a" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 45753 } }
I20260620 18:46:34.830832 23735 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc7f3bb123c348d6837ab2767cdb349a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:46:34.832630 23742 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc7f3bb123c348d6837ab2767cdb349a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc7f3bb123c348d6837ab2767cdb349a. Latest consensus state: current_term: 1 leader_uuid: "fc7f3bb123c348d6837ab2767cdb349a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc7f3bb123c348d6837ab2767cdb349a" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 45753 } } }
I20260620 18:46:34.832968 23742 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc7f3bb123c348d6837ab2767cdb349a [sys.catalog]: This master's current role is: LEADER
I20260620 18:46:34.833598 23741 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc7f3bb123c348d6837ab2767cdb349a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc7f3bb123c348d6837ab2767cdb349a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc7f3bb123c348d6837ab2767cdb349a" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 45753 } } }
I20260620 18:46:34.833879 23741 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc7f3bb123c348d6837ab2767cdb349a [sys.catalog]: This master's current role is: LEADER
I20260620 18:46:34.834028 23745 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:46:34.841631 23745 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:46:34.852080 23745 catalog_manager.cc:1347] Generated new cluster ID: 954b6ed030a544ea887e3810bba03059
I20260620 18:46:34.852213 23745 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:46:34.865180 23745 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:46:34.866521 23745 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:46:34.878609 23745 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc7f3bb123c348d6837ab2767cdb349a: Generated new TSK 0
I20260620 18:46:34.879487 23745 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:46:34.889808 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.129:0
--local_ip_for_outbound_sockets=127.5.225.129
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:45753
--builtin_ntp_servers=127.5.225.148:43471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:46:35.267988 23759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:46:35.268245 23759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:46:35.268411 23759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:46:35.275857 23759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:46:35.275995 23759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:46:35.276183 23759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.129
I20260620 18:46:35.285295 23759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:43471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:45753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.129
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:46:35.286651 23759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:46:35.288446 23759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:46:35.302387 23764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:35.302464 23765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:35.302464 23767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:46:35.303297 23759 server_base.cc:1034] running on GCE node
I20260620 18:46:35.303956 23759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:46:35.304895 23759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:46:35.306098 23759 hybrid_clock.cc:648] HybridClock initialized: now 1781981195306055 us; error 32 us; skew 500 ppm
I20260620 18:46:35.306540 23759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:46:35.308822 23759 webserver.cc:460] Webserver started at http://127.5.225.129:35399/ using document root <none> and password file <none>
I20260620 18:46:35.309535 23759 fs_manager.cc:359] Metadata directory not provided
I20260620 18:46:35.309648 23759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:46:35.309981 23759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:46:35.312800 23759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-0/data/instance:
uuid: "25ebe54cd7864c34a1883cc0e7ee8b77"
format_stamp: "Formatted at 2026-06-20 18:46:35 on dist-test-slave-k6lg"
server_key: "e9b69cc4c312a29812571829aa97546c"
server_key_iv: "682a5d21445ff9e5689367849c69d160"
server_key_version: "encryptionkey@0"
I20260620 18:46:35.313681 23759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance:
uuid: "25ebe54cd7864c34a1883cc0e7ee8b77"
format_stamp: "Formatted at 2026-06-20 18:46:35 on dist-test-slave-k6lg"
server_key: "e9b69cc4c312a29812571829aa97546c"
server_key_iv: "682a5d21445ff9e5689367849c69d160"
server_key_version: "encryptionkey@0"
I20260620 18:46:35.319344 23759 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 18:46:35.322714 23773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:35.324051 23759 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:46:35.324224 23759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-0/wal
uuid: "25ebe54cd7864c34a1883cc0e7ee8b77"
format_stamp: "Formatted at 2026-06-20 18:46:35 on dist-test-slave-k6lg"
server_key: "e9b69cc4c312a29812571829aa97546c"
server_key_iv: "682a5d21445ff9e5689367849c69d160"
server_key_version: "encryptionkey@0"
I20260620 18:46:35.324416 23759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:46:35.341898 23759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:46:35.342921 23759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:46:35.343294 23759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:46:35.344341 23759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:46:35.345840 23759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:46:35.345925 23759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:35.346031 23759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:46:35.346138 23759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:35.379911 23759 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.129:41153
I20260620 18:46:35.379997 23885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.129:41153 every 8 connection(s)
I20260620 18:46:35.381271 23759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
I20260620 18:46:35.387948 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 23759
I20260620 18:46:35.388131 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance
I20260620 18:46:35.388582 6022 external_mini_cluster.cc:1428] Setting key c39cb6eee93888b2387d320380bd7e46
I20260620 18:46:35.391492 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.130:0
--local_ip_for_outbound_sockets=127.5.225.130
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:45753
--builtin_ntp_servers=127.5.225.148:43471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:46:35.394486 23888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:35.383778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:51891 (local address 127.5.225.190:45753)
0620 18:46:35.384225 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:35.384229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:35.385294 (+ 1065us) server_negotiation.cc:407] Connection header received
0620 18:46:35.387131 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:35.387138 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:35.387232 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:35.387349 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:35.388509 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:35.389405 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:35.391428 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:35.391921 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:35.392025 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:35.393620 (+ 1595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:35.393625 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:35.393628 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:46:35.393645 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:35.393682 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:46:35.393694 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:35.393765 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:35.394018 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 18:46:35.394098 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":249,"threads_started":1}
I20260620 18:46:35.395860 23889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:35.384270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45753 (local address 127.5.225.129:51891)
0620 18:46:35.385138 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:35.385180 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:35.386866 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:35.387572 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:35.387580 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:35.387685 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:46:35.388345 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:35.388363 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:35.389580 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:35.389584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:35.391281 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:35.391299 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:35.392147 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:35.392157 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:35.392490 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:35.393266 (+ 776us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:46:35.393272 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:46:35.393326 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:46:35.393430 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:46:35.393794 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:46:35.393808 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:46:35.393819 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 18:46:35.394001 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 18:46:35.394467 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":267,"threads_started":1}
I20260620 18:46:35.399945 23886 heartbeater.cc:347] Connected to a master server at 127.5.225.190:45753
I20260620 18:46:35.400300 23886 heartbeater.cc:464] Registering TS with master...
I20260620 18:46:35.401140 23886 heartbeater.cc:511] Master 127.5.225.190:45753 requested a full tablet report, sending...
I20260620 18:46:35.403364 23700 ts_manager.cc:194] Registered new tserver with Master: 25ebe54cd7864c34a1883cc0e7ee8b77 (127.5.225.129:41153)
I20260620 18:46:35.404870 23700 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.225.129:51891
W20260620 18:46:35.715911 23890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:46:35.716238 23890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:46:35.716401 23890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:46:35.723841 23890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:46:35.723978 23890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:46:35.724151 23890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.130
I20260620 18:46:35.733712 23890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:43471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:45753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.130
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:46:35.735092 23890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:46:35.736934 23890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:46:35.750201 23896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:35.750133 23898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:35.750687 23895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:46:35.751242 23890 server_base.cc:1034] running on GCE node
I20260620 18:46:35.751827 23890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:46:35.752787 23890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:46:35.754016 23890 hybrid_clock.cc:648] HybridClock initialized: now 1781981195753952 us; error 78 us; skew 500 ppm
I20260620 18:46:35.754379 23890 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:46:35.756632 23890 webserver.cc:460] Webserver started at http://127.5.225.130:41071/ using document root <none> and password file <none>
I20260620 18:46:35.757328 23890 fs_manager.cc:359] Metadata directory not provided
I20260620 18:46:35.757450 23890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:46:35.757787 23890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:46:35.760682 23890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-1/data/instance:
uuid: "b89daed9ab9a44cabc157991bf1ffb8a"
format_stamp: "Formatted at 2026-06-20 18:46:35 on dist-test-slave-k6lg"
server_key: "2e6bd7377192d3b14aa51aae4eea923e"
server_key_iv: "bafe48782d6f3bcce67c89bac45fc050"
server_key_version: "encryptionkey@0"
I20260620 18:46:35.761479 23890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance:
uuid: "b89daed9ab9a44cabc157991bf1ffb8a"
format_stamp: "Formatted at 2026-06-20 18:46:35 on dist-test-slave-k6lg"
server_key: "2e6bd7377192d3b14aa51aae4eea923e"
server_key_iv: "bafe48782d6f3bcce67c89bac45fc050"
server_key_version: "encryptionkey@0"
I20260620 18:46:35.766525 23890 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 18:46:35.769964 23904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:35.771333 23890 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:46:35.771512 23890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-1/wal
uuid: "b89daed9ab9a44cabc157991bf1ffb8a"
format_stamp: "Formatted at 2026-06-20 18:46:35 on dist-test-slave-k6lg"
server_key: "2e6bd7377192d3b14aa51aae4eea923e"
server_key_iv: "bafe48782d6f3bcce67c89bac45fc050"
server_key_version: "encryptionkey@0"
I20260620 18:46:35.771746 23890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:46:35.806221 23890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:46:35.807060 23890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:46:35.807372 23890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:46:35.808393 23890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:46:35.809788 23890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:46:35.809921 23890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:35.810057 23890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:46:35.810145 23890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:35.843945 23890 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.130:40937
I20260620 18:46:35.844036 24016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.130:40937 every 8 connection(s)
I20260620 18:46:35.845295 23890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
I20260620 18:46:35.849342 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 23890
I20260620 18:46:35.849509 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance
I20260620 18:46:35.850091 6022 external_mini_cluster.cc:1428] Setting key 0441fd1d5bb8f99b608f308464c0b814
I20260620 18:46:35.853040 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.131:0
--local_ip_for_outbound_sockets=127.5.225.131
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:45753
--builtin_ntp_servers=127.5.225.148:43471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:46:35.858489 23888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:35.848266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:42013 (local address 127.5.225.190:45753)
0620 18:46:35.848481 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:35.848486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:35.849985 (+ 1499us) server_negotiation.cc:407] Connection header received
0620 18:46:35.851984 (+ 1999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:35.851990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:35.852078 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:35.852190 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:35.853307 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:35.854007 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:35.855700 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:35.856110 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:35.856250 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:35.857614 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:35.857631 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:35.857635 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:46:35.857652 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:35.857683 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:46:35.857695 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:35.857789 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:35.858024 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 18:46:35.858135 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 18:46:35.859514 24019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:35.848742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45753 (local address 127.5.225.130:42013)
0620 18:46:35.849798 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:35.849841 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:35.851713 (+ 1872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:35.852425 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:35.852433 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:35.852519 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:46:35.853070 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:35.853087 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:35.854260 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:35.854266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:35.855516 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:35.855529 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:35.856347 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:35.856354 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:35.856574 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:35.857294 (+ 720us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:46:35.857296 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:46:35.857336 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:46:35.857416 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:46:35.857802 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:46:35.857813 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:46:35.857823 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 18:46:35.858008 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 18:46:35.858438 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":351,"threads_started":1}
I20260620 18:46:35.861052 24017 heartbeater.cc:347] Connected to a master server at 127.5.225.190:45753
I20260620 18:46:35.861322 24017 heartbeater.cc:464] Registering TS with master...
I20260620 18:46:35.862013 24017 heartbeater.cc:511] Master 127.5.225.190:45753 requested a full tablet report, sending...
I20260620 18:46:35.863425 23700 ts_manager.cc:194] Registered new tserver with Master: b89daed9ab9a44cabc157991bf1ffb8a (127.5.225.130:40937)
I20260620 18:46:35.864236 23700 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.225.130:42013
W20260620 18:46:36.180800 24020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:46:36.181136 24020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:46:36.181306 24020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:46:36.188692 24020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:46:36.188827 24020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:46:36.189004 24020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.131
I20260620 18:46:36.198282 24020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:43471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:45753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.131
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:46:36.199702 24020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:46:36.202126 24020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:46:36.215407 24026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:36.215446 24028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:36.215328 24025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:46:36.216552 24020 server_base.cc:1034] running on GCE node
I20260620 18:46:36.217141 24020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:46:36.218197 24020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:46:36.219412 24020 hybrid_clock.cc:648] HybridClock initialized: now 1781981196219347 us; error 54 us; skew 500 ppm
I20260620 18:46:36.219753 24020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:46:36.222000 24020 webserver.cc:460] Webserver started at http://127.5.225.131:38225/ using document root <none> and password file <none>
I20260620 18:46:36.222743 24020 fs_manager.cc:359] Metadata directory not provided
I20260620 18:46:36.222858 24020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:46:36.223202 24020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:46:36.226040 24020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-2/data/instance:
uuid: "cb71c6a44b704e4c97067ec60b65e60e"
format_stamp: "Formatted at 2026-06-20 18:46:36 on dist-test-slave-k6lg"
server_key: "daffa5470885e93d28627620de72a380"
server_key_iv: "ebc67ebb9244a684557544ff5ace93e1"
server_key_version: "encryptionkey@0"
I20260620 18:46:36.226904 24020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance:
uuid: "cb71c6a44b704e4c97067ec60b65e60e"
format_stamp: "Formatted at 2026-06-20 18:46:36 on dist-test-slave-k6lg"
server_key: "daffa5470885e93d28627620de72a380"
server_key_iv: "ebc67ebb9244a684557544ff5ace93e1"
server_key_version: "encryptionkey@0"
I20260620 18:46:36.232231 24020 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:46:36.235929 24034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:36.237290 24020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:46:36.237469 24020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-2/wal
uuid: "cb71c6a44b704e4c97067ec60b65e60e"
format_stamp: "Formatted at 2026-06-20 18:46:36 on dist-test-slave-k6lg"
server_key: "daffa5470885e93d28627620de72a380"
server_key_iv: "ebc67ebb9244a684557544ff5ace93e1"
server_key_version: "encryptionkey@0"
I20260620 18:46:36.237699 24020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:46:36.254519 24020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:46:36.255370 24020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:46:36.255676 24020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:46:36.256665 24020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:46:36.258106 24020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:46:36.258249 24020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:36.258396 24020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:46:36.258487 24020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:36.292506 24020 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.131:34301
I20260620 18:46:36.292593 24146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.131:34301 every 8 connection(s)
I20260620 18:46:36.293829 24020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
I20260620 18:46:36.299149 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 24020
I20260620 18:46:36.299311 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance
I20260620 18:46:36.299865 6022 external_mini_cluster.cc:1428] Setting key f0d58f6d22afc31702485c0af45889aa
I20260620 18:46:36.307049 23888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:36.296458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:60365 (local address 127.5.225.190:45753)
0620 18:46:36.296638 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:36.296643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:36.298049 (+ 1406us) server_negotiation.cc:407] Connection header received
0620 18:46:36.300469 (+ 2420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:36.300475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:36.300567 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:36.300698 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:36.301886 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:36.302745 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:36.304328 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:36.304736 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:36.304868 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:36.306220 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:36.306227 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:36.306232 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:46:36.306260 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:36.306291 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:46:36.306304 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:36.306399 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:36.306616 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 18:46:36.306716 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 18:46:36.308038 24149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:36.296886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:45753 (local address 127.5.225.131:60365)
0620 18:46:36.297860 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:36.297920 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:36.300100 (+ 2180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:36.300932 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:36.300940 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:36.301026 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:46:36.301718 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:36.301736 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:36.302914 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:36.302918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:36.304169 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:36.304180 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:36.304966 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:36.304973 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:36.305208 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:36.305851 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:46:36.305854 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:46:36.305892 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:46:36.305971 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:46:36.306398 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:46:36.306410 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:46:36.306420 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 18:46:36.306605 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 18:46:36.307005 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":253,"threads_started":1}
I20260620 18:46:36.309283 24147 heartbeater.cc:347] Connected to a master server at 127.5.225.190:45753
I20260620 18:46:36.309589 24147 heartbeater.cc:464] Registering TS with master...
I20260620 18:46:36.310390 24147 heartbeater.cc:511] Master 127.5.225.190:45753 requested a full tablet report, sending...
I20260620 18:46:36.311681 23700 ts_manager.cc:194] Registered new tserver with Master: cb71c6a44b704e4c97067ec60b65e60e (127.5.225.131:34301)
I20260620 18:46:36.312433 23700 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.225.131:60365
I20260620 18:46:36.314936 6022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:46:36.324270 23888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:36.318942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43500 (local address 127.5.225.190:45753)
0620 18:46:36.319138 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:36.319147 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:36.319578 (+ 431us) server_negotiation.cc:407] Connection header received
0620 18:46:36.319728 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:36.319733 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:36.319823 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:36.319903 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:36.320300 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:36.320948 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:36.322255 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:36.322639 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:36.322729 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:36.323635 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:36.323640 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:36.323643 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:46:36.323659 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:36.323688 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:46:36.323700 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:36.323767 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:36.323881 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 18:46:36.323948 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:46:36.327855 23700 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:46:36.335079 23700 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:46:36.360836 24163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:36.350122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.130:40937 (local address 127.0.0.1:47062)
0620 18:46:36.352444 (+ 2322us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:36.352472 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:36.352628 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:36.353241 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:36.353245 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:36.353270 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:46:36.353544 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:36.353551 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:36.355194 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:36.355201 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:36.357617 (+ 2416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:36.357629 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:36.359516 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:36.359526 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:36.359713 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:36.359744 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 18:46:36.359806 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 18:46:36.359947 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2055,"mutex_wait_us":94,"thread_start_us":1792,"threads_started":1}
I20260620 18:46:36.361227 24164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:36.350391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47062 (local address 127.5.225.130:40937)
0620 18:46:36.352589 (+ 2198us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:36.352599 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:36.352630 (+ 31us) server_negotiation.cc:407] Connection header received
0620 18:46:36.352788 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:36.352798 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:36.353027 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:36.353695 (+ 668us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:46:36.353874 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:36.355010 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:36.357809 (+ 2799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:36.359279 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:36.360442 (+ 1163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:36.360690 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:36.360746 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 18:46:36.360884 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2034,"thread_start_us":1648,"threads_started":1}
I20260620 18:46:36.361230 24161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:36.347227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58294 (local address 127.5.225.129:41153)
0620 18:46:36.347828 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:36.347838 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:36.349589 (+ 1751us) server_negotiation.cc:407] Connection header received
0620 18:46:36.349741 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:36.349749 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:36.350310 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:36.350544 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:46:36.352945 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:36.354659 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:36.358047 (+ 3388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:36.359457 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:36.359595 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:36.359848 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:36.360691 (+ 843us) server_negotiation.cc:299] Negotiation successful
0620 18:46:36.360866 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":713,"threads_started":1}
I20260620 18:46:36.361384 24160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:36.347221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:41153 (local address 127.0.0.1:58294)
0620 18:46:36.349431 (+ 2210us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:36.349457 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:36.349597 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:36.350867 (+ 1270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:36.350878 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:36.350971 (+ 93us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:46:36.352192 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:36.352206 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:36.355220 (+ 3014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:36.355225 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:36.357846 (+ 2621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:36.357861 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:36.359969 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:36.359976 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:36.360150 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:36.360179 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 18:46:36.360275 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 18:46:36.360422 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2030,"mutex_wait_us":126,"spinlock_wait_cycles":2432,"thread_start_us":1821,"threads_started":1}
I20260620 18:46:36.361528 24159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:36.347201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.131:34301 (local address 127.0.0.1:60830)
0620 18:46:36.348444 (+ 1243us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:36.348508 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:36.348717 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:36.350874 (+ 2157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:36.350881 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:36.350987 (+ 106us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:46:36.352861 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:36.352873 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:36.354806 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:36.354811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:36.357857 (+ 3046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:36.357866 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:36.359243 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:36.359249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:36.359437 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:36.359493 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 18:46:36.359657 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 18:46:36.359857 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":926,"mutex_wait_us":792,"spinlock_wait_cycles":88960,"thread_start_us":775,"threads_started":1}
I20260620 18:46:36.362018 24162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:36.347675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60830 (local address 127.5.225.131:34301)
0620 18:46:36.350228 (+ 2553us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:36.350239 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:36.350274 (+ 35us) server_negotiation.cc:407] Connection header received
0620 18:46:36.350365 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:36.350374 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:36.350618 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:36.351354 (+ 736us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:46:36.353048 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:36.354662 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:36.358115 (+ 3453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:36.359055 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:36.359219 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:36.359509 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:36.359701 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 18:46:36.359869 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2357,"thread_start_us":2214,"threads_started":1}
I20260620 18:46:36.364919 23952 tablet_service.cc:1467] Processing CreateTablet for tablet a034b9c5b5a640c68d1021ade0ebb5ab (DEFAULT_TABLE table=test-table [id=83bb4a7b334f41b99c1ed56c848ccf19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:46:36.364987 24082 tablet_service.cc:1467] Processing CreateTablet for tablet a034b9c5b5a640c68d1021ade0ebb5ab (DEFAULT_TABLE table=test-table [id=83bb4a7b334f41b99c1ed56c848ccf19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:46:36.365484 23821 tablet_service.cc:1467] Processing CreateTablet for tablet a034b9c5b5a640c68d1021ade0ebb5ab (DEFAULT_TABLE table=test-table [id=83bb4a7b334f41b99c1ed56c848ccf19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:46:36.366381 23952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a034b9c5b5a640c68d1021ade0ebb5ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:46:36.366393 24082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a034b9c5b5a640c68d1021ade0ebb5ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:46:36.366696 23821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a034b9c5b5a640c68d1021ade0ebb5ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:46:36.377382 24165 tablet_bootstrap.cc:492] T a034b9c5b5a640c68d1021ade0ebb5ab P cb71c6a44b704e4c97067ec60b65e60e: Bootstrap starting.
I20260620 18:46:36.380620 24166 tablet_bootstrap.cc:492] T a034b9c5b5a640c68d1021ade0ebb5ab P b89daed9ab9a44cabc157991bf1ffb8a: Bootstrap starting.
I20260620 18:46:36.381795 24165 tablet_bootstrap.cc:654] T a034b9c5b5a640c68d1021ade0ebb5ab P cb71c6a44b704e4c97067ec60b65e60e: Neither blocks nor log segments found. Creating new log.
I20260620 18:46:36.383155 24165 log.cc:824] T a034b9c5b5a640c68d1021ade0ebb5ab P cb71c6a44b704e4c97067ec60b65e60e: Log is configured to *not* fsync() on all Append() calls
I20260620 18:46:36.384234 24166 tablet_bootstrap.cc:654] T a034b9c5b5a640c68d1021ade0ebb5ab P b89daed9ab9a44cabc157991bf1ffb8a: Neither blocks nor log segments found. Creating new log.
I20260620 18:46:36.385447 24166 log.cc:824] T a034b9c5b5a640c68d1021ade0ebb5ab P b89daed9ab9a44cabc157991bf1ffb8a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:46:36.386379 24165 tablet_bootstrap.cc:492] T a034b9c5b5a640c68d1021ade0ebb5ab P cb71c6a44b704e4c97067ec60b65e60e: No bootstrap required, opened a new log
I20260620 18:46:36.386708 24165 ts_tablet_manager.cc:1397] T a034b9c5b5a640c68d1021ade0ebb5ab P cb71c6a44b704e4c97067ec60b65e60e: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 18:46:36.388942 24168 tablet_bootstrap.cc:492] T a034b9c5b5a640c68d1021ade0ebb5ab P 25ebe54cd7864c34a1883cc0e7ee8b77: Bootstrap starting.
I20260620 18:46:36.391856 24166 tablet_bootstrap.cc:492] T a034b9c5b5a640c68d1021ade0ebb5ab P b89daed9ab9a44cabc157991bf1ffb8a: No bootstrap required, opened a new log
I20260620 18:46:36.392138 24166 ts_tablet_manager.cc:1397] T a034b9c5b5a640c68d1021ade0ebb5ab P b89daed9ab9a44cabc157991bf1ffb8a: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260620 18:46:36.392606 24168 tablet_bootstrap.cc:654] T a034b9c5b5a640c68d1021ade0ebb5ab P 25ebe54cd7864c34a1883cc0e7ee8b77: Neither blocks nor log segments found. Creating new log.
I20260620 18:46:36.393735 24168 log.cc:824] T a034b9c5b5a640c68d1021ade0ebb5ab P 25ebe54cd7864c34a1883cc0e7ee8b77: Log is configured to *not* fsync() on all Append() calls
I20260620 18:46:36.396581 24168 tablet_bootstrap.cc:492] T a034b9c5b5a640c68d1021ade0ebb5ab P 25ebe54cd7864c34a1883cc0e7ee8b77: No bootstrap required, opened a new log
I20260620 18:46:36.396835 24168 ts_tablet_manager.cc:1397] T a034b9c5b5a640c68d1021ade0ebb5ab P 25ebe54cd7864c34a1883cc0e7ee8b77: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260620 18:46:36.397102 24165 raft_consensus.cc:348] T a034b9c5b5a640c68d1021ade0ebb5ab P cb71c6a44b704e4c97067ec60b65e60e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ebe54cd7864c34a1883cc0e7ee8b77" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 41153 } } peers { permanent_uuid: "cb71c6a44b704e4c97067ec60b65e60e" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 34301 } } peers { permanent_uuid: "b89daed9ab9a44cabc157991bf1ffb8a" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40937 } }
I20260620 18:46:36.397540 24165 raft_consensus.cc:374] T a034b9c5b5a640c68d1021ade0ebb5ab P cb71c6a44b704e4c97067ec60b65e60e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:46:36.397641 24165 raft_consensus.cc:729] T a034b9c5b5a640c68d1021ade0ebb5ab P cb71c6a44b704e4c97067ec60b65e60e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb71c6a44b704e4c97067ec60b65e60e, State: Initialized, Role: FOLLOWER
I20260620 18:46:36.398365 24165 consensus_queue.cc:260] T a034b9c5b5a640c68d1021ade0ebb5ab P cb71c6a44b704e4c97067ec60b65e60e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ebe54cd7864c34a1883cc0e7ee8b77" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 41153 } } peers { permanent_uuid: "cb71c6a44b704e4c97067ec60b65e60e" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 34301 } } peers { permanent_uuid: "b89daed9ab9a44cabc157991bf1ffb8a" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40937 } }
I20260620 18:46:36.399590 24147 heartbeater.cc:503] Master 127.5.225.190:45753 was elected leader, sending a full tablet report...
I20260620 18:46:36.400192 24165 ts_tablet_manager.cc:1428] T a034b9c5b5a640c68d1021ade0ebb5ab P cb71c6a44b704e4c97067ec60b65e60e: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260620 18:46:36.400352 24166 raft_consensus.cc:348] T a034b9c5b5a640c68d1021ade0ebb5ab P b89daed9ab9a44cabc157991bf1ffb8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ebe54cd7864c34a1883cc0e7ee8b77" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 41153 } } peers { permanent_uuid: "cb71c6a44b704e4c97067ec60b65e60e" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 34301 } } peers { permanent_uuid: "b89daed9ab9a44cabc157991bf1ffb8a" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40937 } }
I20260620 18:46:36.400753 24166 raft_consensus.cc:374] T a034b9c5b5a640c68d1021ade0ebb5ab P b89daed9ab9a44cabc157991bf1ffb8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:46:36.401057 24166 raft_consensus.cc:729] T a034b9c5b5a640c68d1021ade0ebb5ab P b89daed9ab9a44cabc157991bf1ffb8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89daed9ab9a44cabc157991bf1ffb8a, State: Initialized, Role: FOLLOWER
I20260620 18:46:36.401686 24166 consensus_queue.cc:260] T a034b9c5b5a640c68d1021ade0ebb5ab P b89daed9ab9a44cabc157991bf1ffb8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ebe54cd7864c34a1883cc0e7ee8b77" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 41153 } } peers { permanent_uuid: "cb71c6a44b704e4c97067ec60b65e60e" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 34301 } } peers { permanent_uuid: "b89daed9ab9a44cabc157991bf1ffb8a" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40937 } }
I20260620 18:46:36.402755 24017 heartbeater.cc:503] Master 127.5.225.190:45753 was elected leader, sending a full tablet report...
I20260620 18:46:36.403455 24166 ts_tablet_manager.cc:1428] T a034b9c5b5a640c68d1021ade0ebb5ab P b89daed9ab9a44cabc157991bf1ffb8a: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260620 18:46:36.407210 24168 raft_consensus.cc:348] T a034b9c5b5a640c68d1021ade0ebb5ab P 25ebe54cd7864c34a1883cc0e7ee8b77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ebe54cd7864c34a1883cc0e7ee8b77" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 41153 } } peers { permanent_uuid: "cb71c6a44b704e4c97067ec60b65e60e" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 34301 } } peers { permanent_uuid: "b89daed9ab9a44cabc157991bf1ffb8a" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40937 } }
I20260620 18:46:36.407549 24168 raft_consensus.cc:374] T a034b9c5b5a640c68d1021ade0ebb5ab P 25ebe54cd7864c34a1883cc0e7ee8b77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:46:36.407665 24168 raft_consensus.cc:729] T a034b9c5b5a640c68d1021ade0ebb5ab P 25ebe54cd7864c34a1883cc0e7ee8b77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25ebe54cd7864c34a1883cc0e7ee8b77, State: Initialized, Role: FOLLOWER
I20260620 18:46:36.407898 23886 heartbeater.cc:503] Master 127.5.225.190:45753 was elected leader, sending a full tablet report...
I20260620 18:46:36.408278 24168 consensus_queue.cc:260] T a034b9c5b5a640c68d1021ade0ebb5ab P 25ebe54cd7864c34a1883cc0e7ee8b77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ebe54cd7864c34a1883cc0e7ee8b77" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 41153 } } peers { permanent_uuid: "cb71c6a44b704e4c97067ec60b65e60e" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 34301 } } peers { permanent_uuid: "b89daed9ab9a44cabc157991bf1ffb8a" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40937 } }
I20260620 18:46:36.409830 24168 ts_tablet_manager.cc:1428] T a034b9c5b5a640c68d1021ade0ebb5ab P 25ebe54cd7864c34a1883cc0e7ee8b77: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
I20260620 18:46:36.466887 24173 raft_consensus.cc:482] T a034b9c5b5a640c68d1021ade0ebb5ab P 25ebe54cd7864c34a1883cc0e7ee8b77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:46:36.467303 24173 raft_consensus.cc:504] T a034b9c5b5a640c68d1021ade0ebb5ab P 25ebe54cd7864c34a1883cc0e7ee8b77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ebe54cd7864c34a1883cc0e7ee8b77" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 41153 } } peers { permanent_uuid: "cb71c6a44b704e4c97067ec60b65e60e" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 34301 } } peers { permanent_uuid: "b89daed9ab9a44cabc157991bf1ffb8a" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40937 } }
I20260620 18:46:36.469144 24173 leader_election.cc:288] T a034b9c5b5a640c68d1021ade0ebb5ab P 25ebe54cd7864c34a1883cc0e7ee8b77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb71c6a44b704e4c97067ec60b65e60e (127.5.225.131:34301), b89daed9ab9a44cabc157991bf1ffb8a (127.5.225.130:40937)
I20260620 18:46:36.476346 24162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:36.469464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:32813 (local address 127.5.225.131:34301)
0620 18:46:36.469692 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:36.469698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:36.470260 (+ 562us) server_negotiation.cc:407] Connection header received
0620 18:46:36.470580 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:36.470587 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:36.470676 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:36.470731 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:46:36.471616 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:36.472372 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:36.474364 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:36.475352 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:36.475518 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:36.475596 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:36.475953 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 18:46:36.476098 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:46:36.476466 24174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:36.469316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.131:34301 (local address 127.5.225.129:32813)
0620 18:46:36.470129 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:36.470150 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:36.470390 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:36.470929 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:36.470935 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:36.470959 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:46:36.471451 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:36.471465 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:36.472512 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:36.472519 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:36.474126 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:36.474138 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:36.475551 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:36.475559 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:36.475752 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:36.475789 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 18:46:36.475893 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 18:46:36.476015 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"mutex_wait_us":1,"thread_start_us":332,"threads_started":1}
I20260620 18:46:36.476620 24164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:36.470109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:55355 (local address 127.5.225.130:40937)
0620 18:46:36.470346 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:36.470351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:36.470370 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:46:36.470611 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:36.470618 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:36.470709 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:36.470774 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:46:36.471736 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:36.472448 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:36.474623 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:36.475584 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:36.475752 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:36.475816 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:36.476191 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 18:46:36.476323 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 18:46:36.476804 24175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:36.469522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.130:40937 (local address 127.5.225.129:55355)
0620 18:46:36.470267 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:36.470285 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:36.470422 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:36.470975 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:36.470979 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:36.470998 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:46:36.471552 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:36.471569 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:36.472597 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:36.472601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:36.474414 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:36.474429 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:36.475758 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:36.475762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:36.475923 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:36.475956 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 18:46:36.476144 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 18:46:36.476268 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"mutex_wait_us":36,"thread_start_us":858,"threads_started":1}
I20260620 18:46:36.477691 24102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a034b9c5b5a640c68d1021ade0ebb5ab" candidate_uuid: "25ebe54cd7864c34a1883cc0e7ee8b77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb71c6a44b704e4c97067ec60b65e60e" is_pre_election: true
I20260620 18:46:36.477701 23972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a034b9c5b5a640c68d1021ade0ebb5ab" candidate_uuid: "25ebe54cd7864c34a1883cc0e7ee8b77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89daed9ab9a44cabc157991bf1ffb8a" is_pre_election: true
I20260620 18:46:36.478253 24102 raft_consensus.cc:2454] T a034b9c5b5a640c68d1021ade0ebb5ab P cb71c6a44b704e4c97067ec60b65e60e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25ebe54cd7864c34a1883cc0e7ee8b77 in term 0.
I20260620 18:46:36.478253 23972 raft_consensus.cc:2454] T a034b9c5b5a640c68d1021ade0ebb5ab P b89daed9ab9a44cabc157991bf1ffb8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25ebe54cd7864c34a1883cc0e7ee8b77 in term 0.
I20260620 18:46:36.478924 23775 leader_election.cc:302] T a034b9c5b5a640c68d1021ade0ebb5ab P 25ebe54cd7864c34a1883cc0e7ee8b77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 25ebe54cd7864c34a1883cc0e7ee8b77, b89daed9ab9a44cabc157991bf1ffb8a, cb71c6a44b704e4c97067ec60b65e60e; no voters:
I20260620 18:46:36.479331 24173 raft_consensus.cc:2781] T a034b9c5b5a640c68d1021ade0ebb5ab P 25ebe54cd7864c34a1883cc0e7ee8b77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:46:36.479496 24173 raft_consensus.cc:482] T a034b9c5b5a640c68d1021ade0ebb5ab P 25ebe54cd7864c34a1883cc0e7ee8b77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:46:36.479615 24173 raft_consensus.cc:3037] T a034b9c5b5a640c68d1021ade0ebb5ab P 25ebe54cd7864c34a1883cc0e7ee8b77 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:46:36.481777 24173 raft_consensus.cc:504] T a034b9c5b5a640c68d1021ade0ebb5ab P 25ebe54cd7864c34a1883cc0e7ee8b77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ebe54cd7864c34a1883cc0e7ee8b77" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 41153 } } peers { permanent_uuid: "cb71c6a44b704e4c97067ec60b65e60e" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 34301 } } peers { permanent_uuid: "b89daed9ab9a44cabc157991bf1ffb8a" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40937 } }
I20260620 18:46:36.482515 24173 leader_election.cc:288] T a034b9c5b5a640c68d1021ade0ebb5ab P 25ebe54cd7864c34a1883cc0e7ee8b77 [CANDIDATE]: Term 1 election: Requested vote from peers cb71c6a44b704e4c97067ec60b65e60e (127.5.225.131:34301), b89daed9ab9a44cabc157991bf1ffb8a (127.5.225.130:40937)
I20260620 18:46:36.482954 24102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a034b9c5b5a640c68d1021ade0ebb5ab" candidate_uuid: "25ebe54cd7864c34a1883cc0e7ee8b77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb71c6a44b704e4c97067ec60b65e60e"
I20260620 18:46:36.483001 23972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a034b9c5b5a640c68d1021ade0ebb5ab" candidate_uuid: "25ebe54cd7864c34a1883cc0e7ee8b77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89daed9ab9a44cabc157991bf1ffb8a"
I20260620 18:46:36.483172 23972 raft_consensus.cc:3037] T a034b9c5b5a640c68d1021ade0ebb5ab P b89daed9ab9a44cabc157991bf1ffb8a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:46:36.483172 24102 raft_consensus.cc:3037] T a034b9c5b5a640c68d1021ade0ebb5ab P cb71c6a44b704e4c97067ec60b65e60e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:46:36.486440 24102 raft_consensus.cc:2454] T a034b9c5b5a640c68d1021ade0ebb5ab P cb71c6a44b704e4c97067ec60b65e60e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25ebe54cd7864c34a1883cc0e7ee8b77 in term 1.
I20260620 18:46:36.486459 23972 raft_consensus.cc:2454] T a034b9c5b5a640c68d1021ade0ebb5ab P b89daed9ab9a44cabc157991bf1ffb8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25ebe54cd7864c34a1883cc0e7ee8b77 in term 1.
I20260620 18:46:36.486850 23774 leader_election.cc:302] T a034b9c5b5a640c68d1021ade0ebb5ab P 25ebe54cd7864c34a1883cc0e7ee8b77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25ebe54cd7864c34a1883cc0e7ee8b77, cb71c6a44b704e4c97067ec60b65e60e; no voters:
I20260620 18:46:36.487179 24173 raft_consensus.cc:2781] T a034b9c5b5a640c68d1021ade0ebb5ab P 25ebe54cd7864c34a1883cc0e7ee8b77 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:46:36.487596 24173 raft_consensus.cc:686] T a034b9c5b5a640c68d1021ade0ebb5ab P 25ebe54cd7864c34a1883cc0e7ee8b77 [term 1 LEADER]: Becoming Leader. State: Replica: 25ebe54cd7864c34a1883cc0e7ee8b77, State: Running, Role: LEADER
I20260620 18:46:36.488065 24173 consensus_queue.cc:237] T a034b9c5b5a640c68d1021ade0ebb5ab P 25ebe54cd7864c34a1883cc0e7ee8b77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ebe54cd7864c34a1883cc0e7ee8b77" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 41153 } } peers { permanent_uuid: "cb71c6a44b704e4c97067ec60b65e60e" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 34301 } } peers { permanent_uuid: "b89daed9ab9a44cabc157991bf1ffb8a" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40937 } }
I20260620 18:46:36.492851 23699 catalog_manager.cc:5495] T a034b9c5b5a640c68d1021ade0ebb5ab P 25ebe54cd7864c34a1883cc0e7ee8b77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 25ebe54cd7864c34a1883cc0e7ee8b77 (127.5.225.129). New cstate: current_term: 1 leader_uuid: "25ebe54cd7864c34a1883cc0e7ee8b77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ebe54cd7864c34a1883cc0e7ee8b77" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 41153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb71c6a44b704e4c97067ec60b65e60e" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 34301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89daed9ab9a44cabc157991bf1ffb8a" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40937 } health_report { overall_health: UNKNOWN } } }
I20260620 18:46:36.512764 24161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:36.507285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45664 (local address 127.5.225.129:41153)
0620 18:46:36.507490 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:36.507496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:36.507514 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:46:36.507763 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:36.507768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:36.507862 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:36.508013 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:46:36.508384 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:36.509085 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:36.510582 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:36.510997 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:36.511143 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:36.511659 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:46:36.512162 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:46:36.512325 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:36.512423 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 18:46:36.512500 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:46:36.521143 24102 raft_consensus.cc:1261] T a034b9c5b5a640c68d1021ade0ebb5ab P cb71c6a44b704e4c97067ec60b65e60e [term 1 FOLLOWER]: Refusing update from remote peer 25ebe54cd7864c34a1883cc0e7ee8b77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:46:36.521145 23972 raft_consensus.cc:1261] T a034b9c5b5a640c68d1021ade0ebb5ab P b89daed9ab9a44cabc157991bf1ffb8a [term 1 FOLLOWER]: Refusing update from remote peer 25ebe54cd7864c34a1883cc0e7ee8b77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:46:36.522192 24176 consensus_queue.cc:1035] T a034b9c5b5a640c68d1021ade0ebb5ab P 25ebe54cd7864c34a1883cc0e7ee8b77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89daed9ab9a44cabc157991bf1ffb8a" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:46:36.522568 24173 consensus_queue.cc:1035] T a034b9c5b5a640c68d1021ade0ebb5ab P 25ebe54cd7864c34a1883cc0e7ee8b77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb71c6a44b704e4c97067ec60b65e60e" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 34301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:46:36.539544 24180 mvcc.cc:205] Tried to move back new op lower bound from 7298994980940357632 to 7298994980818870272. Current Snapshot: MvccSnapshot[applied={T|T < 7298994980940357632 or (T in {7298994980940357632})}]
I20260620 18:46:36.550100 23700 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:46:36.550436 23700 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:46:36.553957 23700 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc7f3bb123c348d6837ab2767cdb349a: Sending DeleteTablet for 3 replicas of tablet a034b9c5b5a640c68d1021ade0ebb5ab
I20260620 18:46:36.555094 23821 tablet_service.cc:1514] Processing DeleteTablet for tablet a034b9c5b5a640c68d1021ade0ebb5ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:46:36 UTC) from {username='slave'} at 127.0.0.1:58294
I20260620 18:46:36.555259 23952 tablet_service.cc:1514] Processing DeleteTablet for tablet a034b9c5b5a640c68d1021ade0ebb5ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:46:36 UTC) from {username='slave'} at 127.0.0.1:47062
I20260620 18:46:36.555897 24082 tablet_service.cc:1514] Processing DeleteTablet for tablet a034b9c5b5a640c68d1021ade0ebb5ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:46:36 UTC) from {username='slave'} at 127.0.0.1:60830
I20260620 18:46:36.562424 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 23759
I20260620 18:46:36.563313 24189 tablet_replica.cc:330] T a034b9c5b5a640c68d1021ade0ebb5ab P 25ebe54cd7864c34a1883cc0e7ee8b77: stopping tablet replica
I20260620 18:46:36.564425 24189 raft_consensus.cc:2229] T a034b9c5b5a640c68d1021ade0ebb5ab P 25ebe54cd7864c34a1883cc0e7ee8b77 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:46:36.565717 24189 raft_consensus.cc:2258] T a034b9c5b5a640c68d1021ade0ebb5ab P 25ebe54cd7864c34a1883cc0e7ee8b77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:46:36.569386 24189 ts_tablet_manager.cc:1905] T a034b9c5b5a640c68d1021ade0ebb5ab P 25ebe54cd7864c34a1883cc0e7ee8b77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:46:36.575300 24191 tablet_replica.cc:330] T a034b9c5b5a640c68d1021ade0ebb5ab P cb71c6a44b704e4c97067ec60b65e60e: stopping tablet replica
I20260620 18:46:36.575743 24191 raft_consensus.cc:2229] T a034b9c5b5a640c68d1021ade0ebb5ab P cb71c6a44b704e4c97067ec60b65e60e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:46:36.576263 24191 raft_consensus.cc:2258] T a034b9c5b5a640c68d1021ade0ebb5ab P cb71c6a44b704e4c97067ec60b65e60e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:46:36.580350 24189 ts_tablet_manager.cc:1918] T a034b9c5b5a640c68d1021ade0ebb5ab P 25ebe54cd7864c34a1883cc0e7ee8b77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:46:36.580574 24189 log.cc:1196] T a034b9c5b5a640c68d1021ade0ebb5ab P 25ebe54cd7864c34a1883cc0e7ee8b77: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-0/wal/wals/a034b9c5b5a640c68d1021ade0ebb5ab
I20260620 18:46:36.581213 24189 ts_tablet_manager.cc:1939] T a034b9c5b5a640c68d1021ade0ebb5ab P 25ebe54cd7864c34a1883cc0e7ee8b77: Deleting consensus metadata
I20260620 18:46:36.583179 23687 catalog_manager.cc:4841] TS 25ebe54cd7864c34a1883cc0e7ee8b77 (127.5.225.129:41153): tablet a034b9c5b5a640c68d1021ade0ebb5ab (table test-table [id=83bb4a7b334f41b99c1ed56c848ccf19]) successfully deleted
I20260620 18:46:36.584079 24191 ts_tablet_manager.cc:1905] T a034b9c5b5a640c68d1021ade0ebb5ab P cb71c6a44b704e4c97067ec60b65e60e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:46:36.584836 24190 tablet_replica.cc:330] T a034b9c5b5a640c68d1021ade0ebb5ab P b89daed9ab9a44cabc157991bf1ffb8a: stopping tablet replica
I20260620 18:46:36.585331 24190 raft_consensus.cc:2229] T a034b9c5b5a640c68d1021ade0ebb5ab P b89daed9ab9a44cabc157991bf1ffb8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:46:36.585898 24190 raft_consensus.cc:2258] T a034b9c5b5a640c68d1021ade0ebb5ab P b89daed9ab9a44cabc157991bf1ffb8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:46:36.586836 24161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:36.564325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45680 (local address 127.5.225.129:41153)
0620 18:46:36.564516 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:36.564524 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:36.577015 (+ 12491us) server_negotiation.cc:407] Connection header received
0620 18:46:36.577510 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:36.577517 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:36.577611 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:36.577704 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:36.578418 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:36.579706 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:36.581818 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:36.582473 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:36.582568 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:36.585041 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:36.585052 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:36.585069 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:46:36.585863 (+ 794us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:36.585935 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:46:36.585964 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:36.586056 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:36.586342 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 18:46:36.586458 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 18:46:36.587982 23881 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:46:36.588580 24190 ts_tablet_manager.cc:1905] T a034b9c5b5a640c68d1021ade0ebb5ab P b89daed9ab9a44cabc157991bf1ffb8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:46:36.593029 24191 ts_tablet_manager.cc:1918] T a034b9c5b5a640c68d1021ade0ebb5ab P cb71c6a44b704e4c97067ec60b65e60e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:46:36.593189 24191 log.cc:1196] T a034b9c5b5a640c68d1021ade0ebb5ab P cb71c6a44b704e4c97067ec60b65e60e: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-2/wal/wals/a034b9c5b5a640c68d1021ade0ebb5ab
I20260620 18:46:36.593674 24191 ts_tablet_manager.cc:1939] T a034b9c5b5a640c68d1021ade0ebb5ab P cb71c6a44b704e4c97067ec60b65e60e: Deleting consensus metadata
I20260620 18:46:36.594631 24190 ts_tablet_manager.cc:1918] T a034b9c5b5a640c68d1021ade0ebb5ab P b89daed9ab9a44cabc157991bf1ffb8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:46:36.594805 24190 log.cc:1196] T a034b9c5b5a640c68d1021ade0ebb5ab P b89daed9ab9a44cabc157991bf1ffb8a: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980894474392-6022-0/minicluster-data/ts-1/wal/wals/a034b9c5b5a640c68d1021ade0ebb5ab
I20260620 18:46:36.595296 24190 ts_tablet_manager.cc:1939] T a034b9c5b5a640c68d1021ade0ebb5ab P b89daed9ab9a44cabc157991bf1ffb8a: Deleting consensus metadata
I20260620 18:46:36.595499 23684 catalog_manager.cc:4841] TS cb71c6a44b704e4c97067ec60b65e60e (127.5.225.131:34301): tablet a034b9c5b5a640c68d1021ade0ebb5ab (table test-table [id=83bb4a7b334f41b99c1ed56c848ccf19]) successfully deleted
I20260620 18:46:36.596722 23685 catalog_manager.cc:4841] TS b89daed9ab9a44cabc157991bf1ffb8a (127.5.225.130:40937): tablet a034b9c5b5a640c68d1021ade0ebb5ab (table test-table [id=83bb4a7b334f41b99c1ed56c848ccf19]) successfully deleted
I20260620 18:46:36.684314 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 23759
I20260620 18:46:36.702910 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 23890
I20260620 18:46:36.714056 24164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:36.703718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59250 (local address 127.5.225.130:40937)
0620 18:46:36.703926 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:36.703933 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:36.703976 (+ 43us) server_negotiation.cc:407] Connection header received
0620 18:46:36.704218 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:36.704223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:36.704318 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:36.704431 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:36.704848 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:36.705592 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:36.706949 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:36.707341 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:36.707473 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:36.712056 (+ 4583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:36.712069 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:36.712088 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:46:36.713078 (+ 990us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:36.713170 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:46:36.713194 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:36.713408 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:36.713579 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 18:46:36.713696 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 18:46:36.714870 24012 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:46:36.807245 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 23890
I20260620 18:46:36.824481 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 24020
I20260620 18:46:36.831542 24162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:36.825377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36400 (local address 127.5.225.131:34301)
0620 18:46:36.825563 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:36.825567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:36.825586 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:46:36.825667 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:36.825671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:36.825758 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:36.825849 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:36.826314 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:36.827082 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:36.828328 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:36.828741 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:36.828847 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:36.829856 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:36.829867 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:36.829898 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:46:36.830760 (+ 862us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:36.830815 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:46:36.830839 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:36.830942 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:36.831129 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 18:46:36.831246 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 18:46:36.832252 24142 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:46:36.922104 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 24020
I20260620 18:46:36.939757 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 23669
I20260620 18:46:36.940871 23730 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:46:37.050933 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 23669
2026-06-20T18:46:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2755 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:46:37 dist-test-slave-k6lg krb5kdc[24199](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:46:37 dist-test-slave-k6lg krb5kdc[24199](info): set up 2 sockets
Jun 20 18:46:37 dist-test-slave-k6lg krb5kdc[24199](info): commencing operation
krb5kdc: starting...
W20260620 18:46:38.174340 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:46:38 dist-test-slave-k6lg krb5kdc[24199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:46:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:46:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
I20260620 18:46:38.413007 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:39509
--webserver_interface=127.5.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:39819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:39509
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:46:38.733726 24212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:46:38.734076 24212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:46:38.734254 24212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:46:38.741608 24212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:46:38.741755 24212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:46:38.741855 24212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:46:38.741957 24212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:46:38.742034 24212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:46:38.751741 24212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:39819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:39509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:39509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:46:38.753103 24212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:46:38.754438 24212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:46:38.766099 24217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:38.766062 24218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:38.766036 24220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:46:38.767042 24212 server_base.cc:1034] running on GCE node
I20260620 18:46:38.767787 24212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:46:38.769147 24212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:46:38.770372 24212 hybrid_clock.cc:648] HybridClock initialized: now 1781981198770305 us; error 49 us; skew 500 ppm
I20260620 18:46:38.770712 24212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:46:38 dist-test-slave-k6lg krb5kdc[24199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981198, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:46:38.773856 24212 init.cc:377] Logged in from keytab as kudu/127.5.225.190@KRBTEST.COM (short username kudu)
I20260620 18:46:38.775345 24212 webserver.cc:460] Webserver started at http://127.5.225.190:43687/ using document root <none> and password file <none>
I20260620 18:46:38.776027 24212 fs_manager.cc:359] Metadata directory not provided
I20260620 18:46:38.776173 24212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:46:38.776484 24212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:46:38.779434 24212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "3da5772478794e0aacc149981296fd65"
format_stamp: "Formatted at 2026-06-20 18:46:38 on dist-test-slave-k6lg"
server_key: "17f4b4dd386aff2924a399b5c5c1ad48"
server_key_iv: "0f764dbbcf2b9f3a5221181469ee71f5"
server_key_version: "encryptionkey@0"
I20260620 18:46:38.780200 24212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "3da5772478794e0aacc149981296fd65"
format_stamp: "Formatted at 2026-06-20 18:46:38 on dist-test-slave-k6lg"
server_key: "17f4b4dd386aff2924a399b5c5c1ad48"
server_key_iv: "0f764dbbcf2b9f3a5221181469ee71f5"
server_key_version: "encryptionkey@0"
I20260620 18:46:38.785135 24212 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260620 18:46:38.788419 24227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:38.789652 24212 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 18:46:38.789839 24212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "3da5772478794e0aacc149981296fd65"
format_stamp: "Formatted at 2026-06-20 18:46:38 on dist-test-slave-k6lg"
server_key: "17f4b4dd386aff2924a399b5c5c1ad48"
server_key_iv: "0f764dbbcf2b9f3a5221181469ee71f5"
server_key_version: "encryptionkey@0"
I20260620 18:46:38.790043 24212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:46:38.807406 24212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:46:38.811480 24212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:46:38.811785 24212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:46:38.830796 24212 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:39509
I20260620 18:46:38.830837 24278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:39509 every 8 connection(s)
I20260620 18:46:38.832221 24212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:46:38.835357 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 24212
I20260620 18:46:38.835597 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:46:38.836066 6022 external_mini_cluster.cc:1428] Setting key 3dde9ef71240d5030e89b39fefeb8762
I20260620 18:46:38.836195 24279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:46:38.851343 24279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3da5772478794e0aacc149981296fd65: Bootstrap starting.
Jun 20 18:46:38 dist-test-slave-k6lg krb5kdc[24199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:46:38.855154 24279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3da5772478794e0aacc149981296fd65: Neither blocks nor log segments found. Creating new log.
I20260620 18:46:38.856566 24279 log.cc:824] T 00000000000000000000000000000000 P 3da5772478794e0aacc149981296fd65: Log is configured to *not* fsync() on all Append() calls
I20260620 18:46:38.860167 24279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3da5772478794e0aacc149981296fd65: No bootstrap required, opened a new log
I20260620 18:46:38.864063 24282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:38.837949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52662 (local address 127.5.225.190:39509)
0620 18:46:38.838798 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:38.838809 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:38.838851 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:46:38.841197 (+ 2346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:38.841221 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:38.841677 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:38.842060 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:38.843910 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:38.845211 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:38.847221 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:38.847969 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:38.848130 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:38.853349 (+ 5219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:38.853360 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:38.853380 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:46:38.853436 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:38.857612 (+ 4176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:38.858770 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:38.858780 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:38.858788 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:38.859048 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:38.859604 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:38.859610 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:38.859613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:38.860425 (+ 812us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:38.860732 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:38.862156 (+ 1424us) server_negotiation.cc:299] Negotiation successful
0620 18:46:38.862675 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":421,"threads_started":1}
I20260620 18:46:38.870478 24279 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3da5772478794e0aacc149981296fd65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da5772478794e0aacc149981296fd65" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39509 } }
I20260620 18:46:38.870820 24279 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3da5772478794e0aacc149981296fd65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:46:38.870890 24279 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3da5772478794e0aacc149981296fd65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3da5772478794e0aacc149981296fd65, State: Initialized, Role: FOLLOWER
I20260620 18:46:38.871423 24279 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3da5772478794e0aacc149981296fd65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da5772478794e0aacc149981296fd65" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39509 } }
I20260620 18:46:38.871634 24279 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3da5772478794e0aacc149981296fd65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:46:38.871734 24279 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3da5772478794e0aacc149981296fd65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:46:38.871851 24279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3da5772478794e0aacc149981296fd65 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:46:38.873970 24279 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3da5772478794e0aacc149981296fd65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da5772478794e0aacc149981296fd65" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39509 } }
I20260620 18:46:38.874462 24279 leader_election.cc:302] T 00000000000000000000000000000000 P 3da5772478794e0aacc149981296fd65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3da5772478794e0aacc149981296fd65; no voters:
I20260620 18:46:38.874977 24279 leader_election.cc:288] T 00000000000000000000000000000000 P 3da5772478794e0aacc149981296fd65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:46:38.875151 24284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3da5772478794e0aacc149981296fd65 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:46:38.876314 24284 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3da5772478794e0aacc149981296fd65 [term 1 LEADER]: Becoming Leader. State: Replica: 3da5772478794e0aacc149981296fd65, State: Running, Role: LEADER
I20260620 18:46:38.876863 24284 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3da5772478794e0aacc149981296fd65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da5772478794e0aacc149981296fd65" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39509 } }
I20260620 18:46:38.877200 24279 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3da5772478794e0aacc149981296fd65 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:46:38.879333 24286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3da5772478794e0aacc149981296fd65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3da5772478794e0aacc149981296fd65. Latest consensus state: current_term: 1 leader_uuid: "3da5772478794e0aacc149981296fd65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da5772478794e0aacc149981296fd65" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39509 } } }
I20260620 18:46:38.879657 24286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3da5772478794e0aacc149981296fd65 [sys.catalog]: This master's current role is: LEADER
I20260620 18:46:38.880234 24285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3da5772478794e0aacc149981296fd65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3da5772478794e0aacc149981296fd65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da5772478794e0aacc149981296fd65" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39509 } } }
I20260620 18:46:38.880525 24289 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:46:38.880527 24285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3da5772478794e0aacc149981296fd65 [sys.catalog]: This master's current role is: LEADER
I20260620 18:46:38.891494 24289 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:46:38.900055 24289 catalog_manager.cc:1347] Generated new cluster ID: 9188e6080c784bc6a491c58846dbe56c
I20260620 18:46:38.900177 24289 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:46:38.927107 24289 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:46:38.929004 24289 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:46:38.937390 24289 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3da5772478794e0aacc149981296fd65: Generated new TSK 0
I20260620 18:46:38.938304 24289 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab.
Entry for principal kudu/127.5.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab.
I20260620 18:46:39.007464 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.129:0
--local_ip_for_outbound_sockets=127.5.225.129
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:39509
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:39819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:46:39.341850 24305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:46:39.342115 24305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:46:39.342288 24305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:46:39.349450 24305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:46:39.349555 24305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:46:39.349757 24305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.129
I20260620 18:46:39.359472 24305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:39819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:39509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.129
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:46:39.360769 24305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:46:39.362279 24305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:46:39.375366 24310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:39.375491 24311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:39.375463 24313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:46:39.376379 24305 server_base.cc:1034] running on GCE node
I20260620 18:46:39.376996 24305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:46:39.378016 24305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:46:39.379246 24305 hybrid_clock.cc:648] HybridClock initialized: now 1781981199379189 us; error 49 us; skew 500 ppm
I20260620 18:46:39.379565 24305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:46:39 dist-test-slave-k6lg krb5kdc[24199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981199, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:46:39.382885 24305 init.cc:377] Logged in from keytab as kudu/127.5.225.129@KRBTEST.COM (short username kudu)
I20260620 18:46:39.384505 24305 webserver.cc:460] Webserver started at http://127.5.225.129:35773/ using document root <none> and password file <none>
I20260620 18:46:39.385198 24305 fs_manager.cc:359] Metadata directory not provided
I20260620 18:46:39.385350 24305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:46:39.385679 24305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:46:39.388885 24305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-0/data/instance:
uuid: "7c58463abab84e58b4772d01294a0c78"
format_stamp: "Formatted at 2026-06-20 18:46:39 on dist-test-slave-k6lg"
server_key: "507404c6b9f511bd7d753b503715442a"
server_key_iv: "64419b1d6e0c65e74c36b53b21ad737b"
server_key_version: "encryptionkey@0"
I20260620 18:46:39.389676 24305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance:
uuid: "7c58463abab84e58b4772d01294a0c78"
format_stamp: "Formatted at 2026-06-20 18:46:39 on dist-test-slave-k6lg"
server_key: "507404c6b9f511bd7d753b503715442a"
server_key_iv: "64419b1d6e0c65e74c36b53b21ad737b"
server_key_version: "encryptionkey@0"
I20260620 18:46:39.395527 24305 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 18:46:39.400234 24320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:39.401785 24305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:46:39.401963 24305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-0/wal
uuid: "7c58463abab84e58b4772d01294a0c78"
format_stamp: "Formatted at 2026-06-20 18:46:39 on dist-test-slave-k6lg"
server_key: "507404c6b9f511bd7d753b503715442a"
server_key_iv: "64419b1d6e0c65e74c36b53b21ad737b"
server_key_version: "encryptionkey@0"
I20260620 18:46:39.402232 24305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:46:39.416782 24305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:46:39.420766 24305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:46:39.421000 24305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:46:39.421880 24305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:46:39.423271 24305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:46:39.423342 24305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:39.423426 24305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:46:39.423473 24305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:39.456595 24305 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.129:35499
I20260620 18:46:39.456671 24432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.129:35499 every 8 connection(s)
I20260620 18:46:39.457935 24305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
I20260620 18:46:39.463219 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 24305
I20260620 18:46:39.463413 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance
I20260620 18:46:39.463858 6022 external_mini_cluster.cc:1428] Setting key 7a5e2eec93df3b97575f117a1d3f6e00
Jun 20 18:46:39 dist-test-slave-k6lg krb5kdc[24199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981199, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:46:39.479254 24435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:39.460641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:45273 (local address 127.5.225.190:39509)
0620 18:46:39.461058 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:39.461063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:39.462109 (+ 1046us) server_negotiation.cc:407] Connection header received
0620 18:46:39.464136 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:39.464142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:39.464229 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:39.464349 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:39.466038 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:39.467159 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:39.468696 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:39.469393 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:39.469534 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:39.472926 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:39.472935 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:39.472942 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:46:39.472990 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:39.475892 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:39.476813 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:39.476818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:39.476823 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:39.476950 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:39.477464 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:39.477468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:39.477470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:39.477771 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:39.477906 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:39.478827 (+ 921us) server_negotiation.cc:299] Negotiation successful
0620 18:46:39.478993 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":198,"threads_started":1}
I20260620 18:46:39.480865 24436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:39.461231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39509 (local address 127.5.225.129:45273)
0620 18:46:39.461918 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:39.461959 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:39.463810 (+ 1851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:39.464593 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:39.464605 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:39.465172 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:46:39.465871 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:39.465891 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:39.467327 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:39.467331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:39.468547 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:39.468559 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:39.469609 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:39.469614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:39.469818 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:39.470547 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:46:39.470569 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:46:39.472704 (+ 2135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:46:39.476031 (+ 3327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:46:39.476050 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:46:39.476069 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:46:39.476674 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:46:39.477063 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:46:39.477069 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:46:39.477076 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:46:39.477352 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:46:39.477899 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:46:39.477909 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:46:39.478443 (+ 534us) client_negotiation.cc:769] Sending connection context
0620 18:46:39.478746 (+ 303us) client_negotiation.cc:240] Negotiation successful
0620 18:46:39.479308 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":272,"threads_started":1}
I20260620 18:46:39.482474 24433 heartbeater.cc:347] Connected to a master server at 127.5.225.190:39509
I20260620 18:46:39.482788 24433 heartbeater.cc:464] Registering TS with master...
I20260620 18:46:39.483494 24433 heartbeater.cc:511] Master 127.5.225.190:39509 requested a full tablet report, sending...
I20260620 18:46:39.485553 24244 ts_manager.cc:194] Registered new tserver with Master: 7c58463abab84e58b4772d01294a0c78 (127.5.225.129:35499)
I20260620 18:46:39.487082 24244 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.129@KRBTEST.COM'} at 127.5.225.129:45273
WARNING: no policy specified for kudu/127.5.225.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab.
Entry for principal kudu/127.5.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab.
I20260620 18:46:39.519183 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.130:0
--local_ip_for_outbound_sockets=127.5.225.130
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:39509
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:39819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:46:39.844369 24439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:46:39.844630 24439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:46:39.844723 24439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:46:39.852300 24439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:46:39.852423 24439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:46:39.852543 24439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.130
I20260620 18:46:39.862617 24439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:39819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:39509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.130
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:46:39.863838 24439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:46:39.865061 24439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:46:39.877872 24447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:39.878101 24445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:39.878120 24444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:46:39.884032 24439 server_base.cc:1034] running on GCE node
I20260620 18:46:39.884740 24439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:46:39.885717 24439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:46:39.886945 24439 hybrid_clock.cc:648] HybridClock initialized: now 1781981199886888 us; error 51 us; skew 500 ppm
I20260620 18:46:39.887265 24439 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:46:39 dist-test-slave-k6lg krb5kdc[24199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981199, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:46:39.890419 24439 init.cc:377] Logged in from keytab as kudu/127.5.225.130@KRBTEST.COM (short username kudu)
I20260620 18:46:39.892079 24439 webserver.cc:460] Webserver started at http://127.5.225.130:34463/ using document root <none> and password file <none>
I20260620 18:46:39.892859 24439 fs_manager.cc:359] Metadata directory not provided
I20260620 18:46:39.892999 24439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:46:39.893309 24439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:46:39.896382 24439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-1/data/instance:
uuid: "00817077dc2d4558a5f6b9cc9a76df10"
format_stamp: "Formatted at 2026-06-20 18:46:39 on dist-test-slave-k6lg"
server_key: "0972fdcce83557a6acde2046e2db64e9"
server_key_iv: "2ece90252530fa62902eb69a41a87bcb"
server_key_version: "encryptionkey@0"
I20260620 18:46:39.897166 24439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance:
uuid: "00817077dc2d4558a5f6b9cc9a76df10"
format_stamp: "Formatted at 2026-06-20 18:46:39 on dist-test-slave-k6lg"
server_key: "0972fdcce83557a6acde2046e2db64e9"
server_key_iv: "2ece90252530fa62902eb69a41a87bcb"
server_key_version: "encryptionkey@0"
I20260620 18:46:39.902314 24439 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 18:46:39.905647 24454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:39.907181 24439 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:46:39.907353 24439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-1/wal
uuid: "00817077dc2d4558a5f6b9cc9a76df10"
format_stamp: "Formatted at 2026-06-20 18:46:39 on dist-test-slave-k6lg"
server_key: "0972fdcce83557a6acde2046e2db64e9"
server_key_iv: "2ece90252530fa62902eb69a41a87bcb"
server_key_version: "encryptionkey@0"
I20260620 18:46:39.907554 24439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:46:39.920938 24439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:46:39.924916 24439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:46:39.925227 24439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:46:39.926275 24439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:46:39.927610 24439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:46:39.927728 24439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:39.927876 24439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:46:39.927949 24439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:39.961412 24439 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.130:40113
I20260620 18:46:39.961455 24566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.130:40113 every 8 connection(s)
I20260620 18:46:39.962879 24439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
I20260620 18:46:39.972908 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 24439
I20260620 18:46:39.973137 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance
I20260620 18:46:39.973718 6022 external_mini_cluster.cc:1428] Setting key 2358d7e6c21f7d8c86f40a6cc8f14ec3
Jun 20 18:46:39 dist-test-slave-k6lg krb5kdc[24199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981199, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:46:39.982779 24435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:39.965295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:55131 (local address 127.5.225.190:39509)
0620 18:46:39.965476 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:39.965480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:39.966679 (+ 1199us) server_negotiation.cc:407] Connection header received
0620 18:46:39.968419 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:39.968423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:39.968492 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:39.968580 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:39.969952 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:39.970695 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:39.972335 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:39.972764 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:39.972864 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:39.976793 (+ 3929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:39.976802 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:39.976807 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:46:39.976849 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:39.979651 (+ 2802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:39.980355 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:39.980368 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:39.980371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:39.980504 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:39.980965 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:39.980977 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:39.980979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:39.981233 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:39.981388 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:39.982188 (+ 800us) server_negotiation.cc:299] Negotiation successful
0620 18:46:39.982421 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:46:39.983733 24569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:39.965782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39509 (local address 127.5.225.130:55131)
0620 18:46:39.966509 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:39.966552 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:39.968176 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:39.968778 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:39.968786 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:39.969266 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:46:39.969808 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:39.969824 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:39.970860 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:39.970864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:39.972122 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:39.972135 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:39.973011 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:39.973020 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:39.973330 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:39.974116 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:46:39.974145 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:46:39.976576 (+ 2431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:46:39.979794 (+ 3218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:46:39.979804 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:46:39.979821 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:46:39.980233 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:46:39.980624 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:46:39.980631 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:46:39.980637 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:46:39.980823 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:46:39.981358 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:46:39.981365 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:46:39.981740 (+ 375us) client_negotiation.cc:769] Sending connection context
0620 18:46:39.982089 (+ 349us) client_negotiation.cc:240] Negotiation successful
0620 18:46:39.982636 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":259,"threads_started":1}
I20260620 18:46:39.985170 24567 heartbeater.cc:347] Connected to a master server at 127.5.225.190:39509
I20260620 18:46:39.985488 24567 heartbeater.cc:464] Registering TS with master...
I20260620 18:46:39.986348 24567 heartbeater.cc:511] Master 127.5.225.190:39509 requested a full tablet report, sending...
I20260620 18:46:39.987973 24244 ts_manager.cc:194] Registered new tserver with Master: 00817077dc2d4558a5f6b9cc9a76df10 (127.5.225.130:40113)
I20260620 18:46:39.989209 24244 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.130@KRBTEST.COM'} at 127.5.225.130:55131
WARNING: no policy specified for kudu/127.5.225.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab.
Entry for principal kudu/127.5.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab.
I20260620 18:46:40.030566 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.131:0
--local_ip_for_outbound_sockets=127.5.225.131
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:39509
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:39819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:46:40.362452 24572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:46:40.362720 24572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:46:40.362811 24572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:46:40.370571 24572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:46:40.370685 24572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:46:40.370800 24572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.131
I20260620 18:46:40.380764 24572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:39819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:39509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.131
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:46:40.381980 24572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:46:40.383332 24572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:46:40.398617 24580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:40.398753 24577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:40.398710 24578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:46:40.399286 24572 server_base.cc:1034] running on GCE node
I20260620 18:46:40.399861 24572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:46:40.400733 24572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:46:40.401958 24572 hybrid_clock.cc:648] HybridClock initialized: now 1781981200401917 us; error 29 us; skew 500 ppm
I20260620 18:46:40.402359 24572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:46:40 dist-test-slave-k6lg krb5kdc[24199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981200, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:46:40.405522 24572 init.cc:377] Logged in from keytab as kudu/127.5.225.131@KRBTEST.COM (short username kudu)
I20260620 18:46:40.407043 24572 webserver.cc:460] Webserver started at http://127.5.225.131:33317/ using document root <none> and password file <none>
I20260620 18:46:40.407738 24572 fs_manager.cc:359] Metadata directory not provided
I20260620 18:46:40.407876 24572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:46:40.408190 24572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:46:40.411110 24572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-2/data/instance:
uuid: "a04984776cf34a77850aea0e7bb39e42"
format_stamp: "Formatted at 2026-06-20 18:46:40 on dist-test-slave-k6lg"
server_key: "3844c053d0dc86fe9295a334221bf4d9"
server_key_iv: "58bd33fd0a9219679689b9cdf6140465"
server_key_version: "encryptionkey@0"
I20260620 18:46:40.411866 24572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance:
uuid: "a04984776cf34a77850aea0e7bb39e42"
format_stamp: "Formatted at 2026-06-20 18:46:40 on dist-test-slave-k6lg"
server_key: "3844c053d0dc86fe9295a334221bf4d9"
server_key_iv: "58bd33fd0a9219679689b9cdf6140465"
server_key_version: "encryptionkey@0"
I20260620 18:46:40.416851 24572 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:46:40.420361 24587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:40.421703 24572 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:46:40.421936 24572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-2/wal
uuid: "a04984776cf34a77850aea0e7bb39e42"
format_stamp: "Formatted at 2026-06-20 18:46:40 on dist-test-slave-k6lg"
server_key: "3844c053d0dc86fe9295a334221bf4d9"
server_key_iv: "58bd33fd0a9219679689b9cdf6140465"
server_key_version: "encryptionkey@0"
I20260620 18:46:40.422147 24572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:46:40.440825 24572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:46:40.445437 24572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:46:40.445737 24572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:46:40.446743 24572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:46:40.448110 24572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:46:40.448220 24572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:40.448392 24572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:46:40.448464 24572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:40.481611 24572 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.131:45873
I20260620 18:46:40.481664 24699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.131:45873 every 8 connection(s)
I20260620 18:46:40.483217 24572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
I20260620 18:46:40.486325 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 24572
I20260620 18:46:40.486513 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance
I20260620 18:46:40.487044 6022 external_mini_cluster.cc:1428] Setting key 126eea79faf6acd4b8bf891e0831def3
I20260620 18:46:40.490211 24433 heartbeater.cc:503] Master 127.5.225.190:39509 was elected leader, sending a full tablet report...
Jun 20 18:46:40 dist-test-slave-k6lg krb5kdc[24199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981200, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:46:40.505108 24702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:40.486214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:43287 (local address 127.5.225.190:39509)
0620 18:46:40.486756 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:40.486760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:40.488054 (+ 1294us) server_negotiation.cc:407] Connection header received
0620 18:46:40.490296 (+ 2242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:40.490301 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:40.490399 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:40.490517 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:40.492196 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:40.493124 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:40.494928 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:40.495327 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:40.495485 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:40.498981 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:40.498987 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:40.498990 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:46:40.499021 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:40.501851 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:40.502703 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:40.502713 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:40.502716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:40.502836 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:40.503285 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:40.503290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:40.503295 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:40.503577 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:40.503743 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:40.504464 (+ 721us) server_negotiation.cc:299] Negotiation successful
0620 18:46:40.504707 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":317,"threads_started":1}
I20260620 18:46:40.506309 24703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:40.486890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39509 (local address 127.5.225.131:43287)
0620 18:46:40.487840 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:40.487947 (+ 107us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:40.489974 (+ 2027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:40.490787 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:40.490800 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:40.491358 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:46:40.492032 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:40.492050 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:40.493278 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:40.493284 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:40.494765 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:40.494783 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:40.495578 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:40.495585 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:40.495818 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:40.496514 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:46:40.496540 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:46:40.498752 (+ 2212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:46:40.502018 (+ 3266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:46:40.502027 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:46:40.502045 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:46:40.502573 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:46:40.502955 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:46:40.502961 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:46:40.502965 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:46:40.503164 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:46:40.503714 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:46:40.503722 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:46:40.504086 (+ 364us) client_negotiation.cc:769] Sending connection context
0620 18:46:40.504354 (+ 268us) client_negotiation.cc:240] Negotiation successful
0620 18:46:40.504911 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":399,"threads_started":1}
I20260620 18:46:40.507922 24700 heartbeater.cc:347] Connected to a master server at 127.5.225.190:39509
I20260620 18:46:40.508317 24700 heartbeater.cc:464] Registering TS with master...
I20260620 18:46:40.509056 24700 heartbeater.cc:511] Master 127.5.225.190:39509 requested a full tablet report, sending...
I20260620 18:46:40.510789 24244 ts_manager.cc:194] Registered new tserver with Master: a04984776cf34a77850aea0e7bb39e42 (127.5.225.131:45873)
I20260620 18:46:40.511786 24244 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.131@KRBTEST.COM'} at 127.5.225.131:43287
I20260620 18:46:40.514602 6022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:46:40.531766 24702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:40.518564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52674 (local address 127.5.225.190:39509)
0620 18:46:40.518720 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:40.518725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:40.519239 (+ 514us) server_negotiation.cc:407] Connection header received
0620 18:46:40.519371 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:40.519374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:40.519435 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:40.519536 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:40.520858 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:40.521621 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:40.522961 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:40.523368 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:40.523472 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:40.526094 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:40.526099 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:40.526102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:46:40.526134 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:40.528925 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:40.529736 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:40.529740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:40.529742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:40.529862 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:40.530415 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:40.530419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:40.530421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:40.530774 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:40.530987 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:40.531381 (+ 394us) server_negotiation.cc:299] Negotiation successful
0620 18:46:40.531533 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 18:46:40.536026 24244 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:46:40.543180 24244 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:46:40.569574 24713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:40.556396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:35499 (local address 127.0.0.1:52448)
0620 18:46:40.558777 (+ 2381us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:40.558797 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:40.558904 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:40.561732 (+ 2828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:40.561740 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:40.561772 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:46:40.562120 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:40.562137 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:40.564405 (+ 2268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:40.564412 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:40.566675 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:40.566687 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:40.568526 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:40.568534 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:40.568729 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:40.568800 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 18:46:40.568910 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 18:46:40.569107 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2253,"mutex_wait_us":56,"thread_start_us":971,"threads_started":1}
I20260620 18:46:40.570073 24718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:40.559801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.130:40113 (local address 127.0.0.1:46128)
0620 18:46:40.561267 (+ 1466us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:40.561290 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:40.561447 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:40.562728 (+ 1281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:40.562733 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:40.562759 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:46:40.563165 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:40.563177 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:40.564898 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:40.564905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:40.566823 (+ 1918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:40.566836 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:40.568890 (+ 2054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:40.568896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:40.569082 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:40.569118 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 18:46:40.569211 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 18:46:40.569342 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1316,"mutex_wait_us":1,"thread_start_us":1171,"threads_started":1}
I20260620 18:46:40.570147 24717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:40.559144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46128 (local address 127.5.225.130:40113)
0620 18:46:40.562005 (+ 2861us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:40.562017 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:40.562049 (+ 32us) server_negotiation.cc:407] Connection header received
0620 18:46:40.562198 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:40.562209 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:40.562506 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:40.562689 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:46:40.563361 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:40.564759 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:40.567234 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:40.568657 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:40.568814 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:40.569003 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:40.569276 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 18:46:40.569450 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2670,"thread_start_us":2556,"threads_started":1}
I20260620 18:46:40.570631 24715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:40.555645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.131:45873 (local address 127.0.0.1:47668)
0620 18:46:40.557829 (+ 2184us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:40.557884 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:40.558141 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:40.563412 (+ 5271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:40.563417 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:40.563442 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:46:40.563841 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:40.563853 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:40.566039 (+ 2186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:40.566044 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:40.567904 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:40.567921 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:40.569789 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:40.569796 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:40.570005 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:40.570046 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 18:46:40.570119 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 18:46:40.570285 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1846,"mutex_wait_us":1,"thread_start_us":2853,"threads_started":1}
I20260620 18:46:40.570693 24716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:40.557163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52448 (local address 127.5.225.129:35499)
0620 18:46:40.560912 (+ 3749us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:40.560924 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:40.560959 (+ 35us) server_negotiation.cc:407] Connection header received
0620 18:46:40.561061 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:40.561070 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:40.561426 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:40.562294 (+ 868us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:46:40.562586 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:40.564242 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:40.566832 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:40.568294 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:40.569818 (+ 1524us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:40.570060 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:40.570147 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 18:46:40.570342 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3523,"thread_start_us":3352,"threads_started":1}
I20260620 18:46:40.571645 24714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:40.555802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47668 (local address 127.5.225.131:45873)
0620 18:46:40.562797 (+ 6995us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:40.562807 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:40.562839 (+ 32us) server_negotiation.cc:407] Connection header received
0620 18:46:40.562955 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:40.562965 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:40.563204 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:40.564036 (+ 832us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:46:40.564314 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:40.565888 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:40.568097 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:40.569551 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:40.570864 (+ 1313us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:40.571114 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:40.571180 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 18:46:40.571337 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6807,"thread_start_us":6663,"threads_started":1}
I20260620 18:46:40.574342 24368 tablet_service.cc:1467] Processing CreateTablet for tablet cf64c285a67043c1962226ed7a391aa4 (DEFAULT_TABLE table=test-table [id=294b4004d2754bee825b50439efabf5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:46:40.574193 24502 tablet_service.cc:1467] Processing CreateTablet for tablet cf64c285a67043c1962226ed7a391aa4 (DEFAULT_TABLE table=test-table [id=294b4004d2754bee825b50439efabf5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:46:40.575537 24368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf64c285a67043c1962226ed7a391aa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:46:40.575611 24502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf64c285a67043c1962226ed7a391aa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:46:40.575225 24635 tablet_service.cc:1467] Processing CreateTablet for tablet cf64c285a67043c1962226ed7a391aa4 (DEFAULT_TABLE table=test-table [id=294b4004d2754bee825b50439efabf5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:46:40.576493 24635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf64c285a67043c1962226ed7a391aa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:46:40.588050 24719 tablet_bootstrap.cc:492] T cf64c285a67043c1962226ed7a391aa4 P 7c58463abab84e58b4772d01294a0c78: Bootstrap starting.
I20260620 18:46:40.591451 24721 tablet_bootstrap.cc:492] T cf64c285a67043c1962226ed7a391aa4 P a04984776cf34a77850aea0e7bb39e42: Bootstrap starting.
I20260620 18:46:40.591688 24719 tablet_bootstrap.cc:654] T cf64c285a67043c1962226ed7a391aa4 P 7c58463abab84e58b4772d01294a0c78: Neither blocks nor log segments found. Creating new log.
I20260620 18:46:40.593043 24719 log.cc:824] T cf64c285a67043c1962226ed7a391aa4 P 7c58463abab84e58b4772d01294a0c78: Log is configured to *not* fsync() on all Append() calls
I20260620 18:46:40.593225 24720 tablet_bootstrap.cc:492] T cf64c285a67043c1962226ed7a391aa4 P 00817077dc2d4558a5f6b9cc9a76df10: Bootstrap starting.
I20260620 18:46:40.595316 24721 tablet_bootstrap.cc:654] T cf64c285a67043c1962226ed7a391aa4 P a04984776cf34a77850aea0e7bb39e42: Neither blocks nor log segments found. Creating new log.
I20260620 18:46:40.596670 24721 log.cc:824] T cf64c285a67043c1962226ed7a391aa4 P a04984776cf34a77850aea0e7bb39e42: Log is configured to *not* fsync() on all Append() calls
I20260620 18:46:40.596763 24720 tablet_bootstrap.cc:654] T cf64c285a67043c1962226ed7a391aa4 P 00817077dc2d4558a5f6b9cc9a76df10: Neither blocks nor log segments found. Creating new log.
I20260620 18:46:40.597039 24719 tablet_bootstrap.cc:492] T cf64c285a67043c1962226ed7a391aa4 P 7c58463abab84e58b4772d01294a0c78: No bootstrap required, opened a new log
I20260620 18:46:40.597368 24719 ts_tablet_manager.cc:1397] T cf64c285a67043c1962226ed7a391aa4 P 7c58463abab84e58b4772d01294a0c78: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 18:46:40.598011 24720 log.cc:824] T cf64c285a67043c1962226ed7a391aa4 P 00817077dc2d4558a5f6b9cc9a76df10: Log is configured to *not* fsync() on all Append() calls
I20260620 18:46:40.599700 24721 tablet_bootstrap.cc:492] T cf64c285a67043c1962226ed7a391aa4 P a04984776cf34a77850aea0e7bb39e42: No bootstrap required, opened a new log
I20260620 18:46:40.599956 24721 ts_tablet_manager.cc:1397] T cf64c285a67043c1962226ed7a391aa4 P a04984776cf34a77850aea0e7bb39e42: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260620 18:46:40.600463 24720 tablet_bootstrap.cc:492] T cf64c285a67043c1962226ed7a391aa4 P 00817077dc2d4558a5f6b9cc9a76df10: No bootstrap required, opened a new log
I20260620 18:46:40.600713 24720 ts_tablet_manager.cc:1397] T cf64c285a67043c1962226ed7a391aa4 P 00817077dc2d4558a5f6b9cc9a76df10: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 18:46:40.607928 24720 raft_consensus.cc:348] T cf64c285a67043c1962226ed7a391aa4 P 00817077dc2d4558a5f6b9cc9a76df10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a04984776cf34a77850aea0e7bb39e42" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 45873 } } peers { permanent_uuid: "00817077dc2d4558a5f6b9cc9a76df10" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40113 } } peers { permanent_uuid: "7c58463abab84e58b4772d01294a0c78" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35499 } }
I20260620 18:46:40.608296 24720 raft_consensus.cc:374] T cf64c285a67043c1962226ed7a391aa4 P 00817077dc2d4558a5f6b9cc9a76df10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:46:40.608393 24720 raft_consensus.cc:729] T cf64c285a67043c1962226ed7a391aa4 P 00817077dc2d4558a5f6b9cc9a76df10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00817077dc2d4558a5f6b9cc9a76df10, State: Initialized, Role: FOLLOWER
I20260620 18:46:40.608479 24719 raft_consensus.cc:348] T cf64c285a67043c1962226ed7a391aa4 P 7c58463abab84e58b4772d01294a0c78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a04984776cf34a77850aea0e7bb39e42" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 45873 } } peers { permanent_uuid: "00817077dc2d4558a5f6b9cc9a76df10" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40113 } } peers { permanent_uuid: "7c58463abab84e58b4772d01294a0c78" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35499 } }
I20260620 18:46:40.608922 24719 raft_consensus.cc:374] T cf64c285a67043c1962226ed7a391aa4 P 7c58463abab84e58b4772d01294a0c78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:46:40.609036 24719 raft_consensus.cc:729] T cf64c285a67043c1962226ed7a391aa4 P 7c58463abab84e58b4772d01294a0c78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c58463abab84e58b4772d01294a0c78, State: Initialized, Role: FOLLOWER
I20260620 18:46:40.609119 24720 consensus_queue.cc:260] T cf64c285a67043c1962226ed7a391aa4 P 00817077dc2d4558a5f6b9cc9a76df10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a04984776cf34a77850aea0e7bb39e42" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 45873 } } peers { permanent_uuid: "00817077dc2d4558a5f6b9cc9a76df10" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40113 } } peers { permanent_uuid: "7c58463abab84e58b4772d01294a0c78" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35499 } }
I20260620 18:46:40.609625 24719 consensus_queue.cc:260] T cf64c285a67043c1962226ed7a391aa4 P 7c58463abab84e58b4772d01294a0c78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a04984776cf34a77850aea0e7bb39e42" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 45873 } } peers { permanent_uuid: "00817077dc2d4558a5f6b9cc9a76df10" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40113 } } peers { permanent_uuid: "7c58463abab84e58b4772d01294a0c78" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35499 } }
I20260620 18:46:40.610385 24567 heartbeater.cc:503] Master 127.5.225.190:39509 was elected leader, sending a full tablet report...
I20260620 18:46:40.610309 24721 raft_consensus.cc:348] T cf64c285a67043c1962226ed7a391aa4 P a04984776cf34a77850aea0e7bb39e42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a04984776cf34a77850aea0e7bb39e42" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 45873 } } peers { permanent_uuid: "00817077dc2d4558a5f6b9cc9a76df10" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40113 } } peers { permanent_uuid: "7c58463abab84e58b4772d01294a0c78" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35499 } }
I20260620 18:46:40.610839 24721 raft_consensus.cc:374] T cf64c285a67043c1962226ed7a391aa4 P a04984776cf34a77850aea0e7bb39e42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:46:40.610957 24721 raft_consensus.cc:729] T cf64c285a67043c1962226ed7a391aa4 P a04984776cf34a77850aea0e7bb39e42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04984776cf34a77850aea0e7bb39e42, State: Initialized, Role: FOLLOWER
I20260620 18:46:40.611662 24721 consensus_queue.cc:260] T cf64c285a67043c1962226ed7a391aa4 P a04984776cf34a77850aea0e7bb39e42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a04984776cf34a77850aea0e7bb39e42" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 45873 } } peers { permanent_uuid: "00817077dc2d4558a5f6b9cc9a76df10" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40113 } } peers { permanent_uuid: "7c58463abab84e58b4772d01294a0c78" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35499 } }
I20260620 18:46:40.612469 24720 ts_tablet_manager.cc:1428] T cf64c285a67043c1962226ed7a391aa4 P 00817077dc2d4558a5f6b9cc9a76df10: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260620 18:46:40.615072 24700 heartbeater.cc:503] Master 127.5.225.190:39509 was elected leader, sending a full tablet report...
I20260620 18:46:40.615656 24721 ts_tablet_manager.cc:1428] T cf64c285a67043c1962226ed7a391aa4 P a04984776cf34a77850aea0e7bb39e42: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260620 18:46:40.617597 24719 ts_tablet_manager.cc:1428] T cf64c285a67043c1962226ed7a391aa4 P 7c58463abab84e58b4772d01294a0c78: Time spent starting tablet: real 0.020s user 0.013s sys 0.005s
I20260620 18:46:40.667953 24725 raft_consensus.cc:482] T cf64c285a67043c1962226ed7a391aa4 P 00817077dc2d4558a5f6b9cc9a76df10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:46:40.668291 24725 raft_consensus.cc:504] T cf64c285a67043c1962226ed7a391aa4 P 00817077dc2d4558a5f6b9cc9a76df10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a04984776cf34a77850aea0e7bb39e42" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 45873 } } peers { permanent_uuid: "00817077dc2d4558a5f6b9cc9a76df10" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40113 } } peers { permanent_uuid: "7c58463abab84e58b4772d01294a0c78" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35499 } }
I20260620 18:46:40.669984 24725 leader_election.cc:288] T cf64c285a67043c1962226ed7a391aa4 P 00817077dc2d4558a5f6b9cc9a76df10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a04984776cf34a77850aea0e7bb39e42 (127.5.225.131:45873), 7c58463abab84e58b4772d01294a0c78 (127.5.225.129:35499)
I20260620 18:46:40.679792 24714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:40.670584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:51619 (local address 127.5.225.131:45873)
0620 18:46:40.670790 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:40.670796 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:40.671574 (+ 778us) server_negotiation.cc:407] Connection header received
0620 18:46:40.671791 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:40.671797 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:40.671878 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:40.671981 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:46:40.672880 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:40.674058 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:40.677300 (+ 3242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:40.678572 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:40.678742 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:40.678815 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:40.679393 (+ 578us) server_negotiation.cc:299] Negotiation successful
0620 18:46:40.679525 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:46:40.679795 24716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:40.670554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:50657 (local address 127.5.225.129:35499)
0620 18:46:40.670805 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:40.670812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:40.671584 (+ 772us) server_negotiation.cc:407] Connection header received
0620 18:46:40.671791 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:40.671797 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:40.671878 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:40.671981 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:46:40.672936 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:40.674057 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:40.677300 (+ 3243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:40.678572 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:40.678738 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:40.678815 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:40.679393 (+ 578us) server_negotiation.cc:299] Negotiation successful
0620 18:46:40.679545 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260620 18:46:40.680085 24728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:40.670449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:35499 (local address 127.5.225.130:50657)
0620 18:46:40.671478 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:40.671494 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:40.671605 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:40.672014 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:40.672019 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:40.672065 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:46:40.672761 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:40.672774 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:40.674216 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:40.674222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:40.677112 (+ 2890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:40.677136 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:40.678856 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:40.678860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:40.679148 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:40.679248 (+ 100us) client_negotiation.cc:769] Sending connection context
0620 18:46:40.679348 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 18:46:40.679511 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":914,"mutex_wait_us":398,"spinlock_wait_cycles":16000,"thread_start_us":559,"threads_started":1}
I20260620 18:46:40.680213 24729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:40.670449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.131:45873 (local address 127.5.225.130:51619)
0620 18:46:40.671416 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:40.671437 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:40.671646 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:40.672014 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:40.672019 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:40.672066 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:46:40.672692 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:40.672710 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:40.674216 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:40.674222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:40.677112 (+ 2890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:40.677135 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:40.678852 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:40.678859 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:40.679149 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:40.679252 (+ 103us) client_negotiation.cc:769] Sending connection context
0620 18:46:40.679348 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 18:46:40.679806 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"mutex_wait_us":389,"spinlock_wait_cycles":47488,"thread_start_us":530,"threads_started":1}
I20260620 18:46:40.681246 24655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf64c285a67043c1962226ed7a391aa4" candidate_uuid: "00817077dc2d4558a5f6b9cc9a76df10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04984776cf34a77850aea0e7bb39e42" is_pre_election: true
I20260620 18:46:40.681259 24388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf64c285a67043c1962226ed7a391aa4" candidate_uuid: "00817077dc2d4558a5f6b9cc9a76df10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c58463abab84e58b4772d01294a0c78" is_pre_election: true
I20260620 18:46:40.681845 24655 raft_consensus.cc:2454] T cf64c285a67043c1962226ed7a391aa4 P a04984776cf34a77850aea0e7bb39e42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00817077dc2d4558a5f6b9cc9a76df10 in term 0.
I20260620 18:46:40.681845 24388 raft_consensus.cc:2454] T cf64c285a67043c1962226ed7a391aa4 P 7c58463abab84e58b4772d01294a0c78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00817077dc2d4558a5f6b9cc9a76df10 in term 0.
I20260620 18:46:40.682610 24455 leader_election.cc:302] T cf64c285a67043c1962226ed7a391aa4 P 00817077dc2d4558a5f6b9cc9a76df10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 00817077dc2d4558a5f6b9cc9a76df10, 7c58463abab84e58b4772d01294a0c78, a04984776cf34a77850aea0e7bb39e42; no voters:
I20260620 18:46:40.683053 24725 raft_consensus.cc:2781] T cf64c285a67043c1962226ed7a391aa4 P 00817077dc2d4558a5f6b9cc9a76df10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:46:40.683211 24725 raft_consensus.cc:482] T cf64c285a67043c1962226ed7a391aa4 P 00817077dc2d4558a5f6b9cc9a76df10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:46:40.683323 24725 raft_consensus.cc:3037] T cf64c285a67043c1962226ed7a391aa4 P 00817077dc2d4558a5f6b9cc9a76df10 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:46:40.685523 24725 raft_consensus.cc:504] T cf64c285a67043c1962226ed7a391aa4 P 00817077dc2d4558a5f6b9cc9a76df10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a04984776cf34a77850aea0e7bb39e42" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 45873 } } peers { permanent_uuid: "00817077dc2d4558a5f6b9cc9a76df10" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40113 } } peers { permanent_uuid: "7c58463abab84e58b4772d01294a0c78" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35499 } }
I20260620 18:46:40.686205 24725 leader_election.cc:288] T cf64c285a67043c1962226ed7a391aa4 P 00817077dc2d4558a5f6b9cc9a76df10 [CANDIDATE]: Term 1 election: Requested vote from peers a04984776cf34a77850aea0e7bb39e42 (127.5.225.131:45873), 7c58463abab84e58b4772d01294a0c78 (127.5.225.129:35499)
I20260620 18:46:40.686641 24655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf64c285a67043c1962226ed7a391aa4" candidate_uuid: "00817077dc2d4558a5f6b9cc9a76df10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04984776cf34a77850aea0e7bb39e42"
I20260620 18:46:40.686631 24388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf64c285a67043c1962226ed7a391aa4" candidate_uuid: "00817077dc2d4558a5f6b9cc9a76df10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c58463abab84e58b4772d01294a0c78"
I20260620 18:46:40.686846 24655 raft_consensus.cc:3037] T cf64c285a67043c1962226ed7a391aa4 P a04984776cf34a77850aea0e7bb39e42 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:46:40.686841 24388 raft_consensus.cc:3037] T cf64c285a67043c1962226ed7a391aa4 P 7c58463abab84e58b4772d01294a0c78 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:46:40.689188 24655 raft_consensus.cc:2454] T cf64c285a67043c1962226ed7a391aa4 P a04984776cf34a77850aea0e7bb39e42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00817077dc2d4558a5f6b9cc9a76df10 in term 1.
I20260620 18:46:40.689256 24388 raft_consensus.cc:2454] T cf64c285a67043c1962226ed7a391aa4 P 7c58463abab84e58b4772d01294a0c78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00817077dc2d4558a5f6b9cc9a76df10 in term 1.
I20260620 18:46:40.689643 24455 leader_election.cc:302] T cf64c285a67043c1962226ed7a391aa4 P 00817077dc2d4558a5f6b9cc9a76df10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00817077dc2d4558a5f6b9cc9a76df10, a04984776cf34a77850aea0e7bb39e42; no voters:
I20260620 18:46:40.689939 24725 raft_consensus.cc:2781] T cf64c285a67043c1962226ed7a391aa4 P 00817077dc2d4558a5f6b9cc9a76df10 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:46:40.690428 24725 raft_consensus.cc:686] T cf64c285a67043c1962226ed7a391aa4 P 00817077dc2d4558a5f6b9cc9a76df10 [term 1 LEADER]: Becoming Leader. State: Replica: 00817077dc2d4558a5f6b9cc9a76df10, State: Running, Role: LEADER
I20260620 18:46:40.690943 24725 consensus_queue.cc:237] T cf64c285a67043c1962226ed7a391aa4 P 00817077dc2d4558a5f6b9cc9a76df10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a04984776cf34a77850aea0e7bb39e42" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 45873 } } peers { permanent_uuid: "00817077dc2d4558a5f6b9cc9a76df10" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40113 } } peers { permanent_uuid: "7c58463abab84e58b4772d01294a0c78" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35499 } }
I20260620 18:46:40.695479 24243 catalog_manager.cc:5495] T cf64c285a67043c1962226ed7a391aa4 P 00817077dc2d4558a5f6b9cc9a76df10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00817077dc2d4558a5f6b9cc9a76df10 (127.5.225.130). New cstate: current_term: 1 leader_uuid: "00817077dc2d4558a5f6b9cc9a76df10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a04984776cf34a77850aea0e7bb39e42" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 45873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00817077dc2d4558a5f6b9cc9a76df10" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c58463abab84e58b4772d01294a0c78" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35499 } health_report { overall_health: UNKNOWN } } }
W20260620 18:46:40.710423 24434 tablet.cc:2369] T cf64c285a67043c1962226ed7a391aa4 P 7c58463abab84e58b4772d01294a0c78: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:46:40.714831 24568 tablet.cc:2369] T cf64c285a67043c1962226ed7a391aa4 P 00817077dc2d4558a5f6b9cc9a76df10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:46:40.723101 24717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:40.717997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46142 (local address 127.5.225.130:40113)
0620 18:46:40.718264 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:40.718269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:40.718287 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:46:40.718476 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:40.718480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:40.718571 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:40.718692 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:46:40.719034 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:40.719744 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:40.721017 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:40.721420 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:40.721512 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:40.722021 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:46:40.722533 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:46:40.722658 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:40.722802 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 18:46:40.722840 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 18:46:40.731323 24388 raft_consensus.cc:1261] T cf64c285a67043c1962226ed7a391aa4 P 7c58463abab84e58b4772d01294a0c78 [term 1 FOLLOWER]: Refusing update from remote peer 00817077dc2d4558a5f6b9cc9a76df10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:46:40.731323 24655 raft_consensus.cc:1261] T cf64c285a67043c1962226ed7a391aa4 P a04984776cf34a77850aea0e7bb39e42 [term 1 FOLLOWER]: Refusing update from remote peer 00817077dc2d4558a5f6b9cc9a76df10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:46:40.732295 24725 consensus_queue.cc:1035] T cf64c285a67043c1962226ed7a391aa4 P 00817077dc2d4558a5f6b9cc9a76df10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c58463abab84e58b4772d01294a0c78" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 35499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:46:40.732712 24730 consensus_queue.cc:1035] T cf64c285a67043c1962226ed7a391aa4 P 00817077dc2d4558a5f6b9cc9a76df10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04984776cf34a77850aea0e7bb39e42" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 45873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 18:46:40.735194 24701 tablet.cc:2369] T cf64c285a67043c1962226ed7a391aa4 P a04984776cf34a77850aea0e7bb39e42: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:46:40.760912 24244 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:46:40.761215 24244 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:46:40.764930 24244 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3da5772478794e0aacc149981296fd65: Sending DeleteTablet for 3 replicas of tablet cf64c285a67043c1962226ed7a391aa4
I20260620 18:46:40.766198 24635 tablet_service.cc:1514] Processing DeleteTablet for tablet cf64c285a67043c1962226ed7a391aa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:46:40 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:47668
I20260620 18:46:40.766301 24502 tablet_service.cc:1514] Processing DeleteTablet for tablet cf64c285a67043c1962226ed7a391aa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:46:40 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:46128
I20260620 18:46:40.767112 24743 tablet_replica.cc:330] T cf64c285a67043c1962226ed7a391aa4 P a04984776cf34a77850aea0e7bb39e42: stopping tablet replica
I20260620 18:46:40.767011 24368 tablet_service.cc:1514] Processing DeleteTablet for tablet cf64c285a67043c1962226ed7a391aa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:46:40 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:52448
I20260620 18:46:40.767638 24743 raft_consensus.cc:2229] T cf64c285a67043c1962226ed7a391aa4 P a04984776cf34a77850aea0e7bb39e42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:46:40.768211 24743 raft_consensus.cc:2258] T cf64c285a67043c1962226ed7a391aa4 P a04984776cf34a77850aea0e7bb39e42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:46:40.770627 24743 ts_tablet_manager.cc:1905] T cf64c285a67043c1962226ed7a391aa4 P a04984776cf34a77850aea0e7bb39e42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:46:40.771054 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 24305
I20260620 18:46:40.778775 24743 ts_tablet_manager.cc:1918] T cf64c285a67043c1962226ed7a391aa4 P a04984776cf34a77850aea0e7bb39e42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:46:40.779090 24743 log.cc:1196] T cf64c285a67043c1962226ed7a391aa4 P a04984776cf34a77850aea0e7bb39e42: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-2/wal/wals/cf64c285a67043c1962226ed7a391aa4
I20260620 18:46:40.779769 24743 ts_tablet_manager.cc:1939] T cf64c285a67043c1962226ed7a391aa4 P a04984776cf34a77850aea0e7bb39e42: Deleting consensus metadata
I20260620 18:46:40.781962 24228 catalog_manager.cc:4841] TS a04984776cf34a77850aea0e7bb39e42 (127.5.225.131:45873): tablet cf64c285a67043c1962226ed7a391aa4 (table test-table [id=294b4004d2754bee825b50439efabf5b]) successfully deleted
I20260620 18:46:40.786324 24745 tablet_replica.cc:330] T cf64c285a67043c1962226ed7a391aa4 P 7c58463abab84e58b4772d01294a0c78: stopping tablet replica
I20260620 18:46:40.786962 24745 raft_consensus.cc:2229] T cf64c285a67043c1962226ed7a391aa4 P 7c58463abab84e58b4772d01294a0c78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:46:40.787606 24745 raft_consensus.cc:2258] T cf64c285a67043c1962226ed7a391aa4 P 7c58463abab84e58b4772d01294a0c78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:46:40.789151 24744 tablet_replica.cc:330] T cf64c285a67043c1962226ed7a391aa4 P 00817077dc2d4558a5f6b9cc9a76df10: stopping tablet replica
I20260620 18:46:40.789674 24744 raft_consensus.cc:2229] T cf64c285a67043c1962226ed7a391aa4 P 00817077dc2d4558a5f6b9cc9a76df10 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:46:40.790254 24745 ts_tablet_manager.cc:1905] T cf64c285a67043c1962226ed7a391aa4 P 7c58463abab84e58b4772d01294a0c78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:46:40.790481 24744 raft_consensus.cc:2258] T cf64c285a67043c1962226ed7a391aa4 P 00817077dc2d4558a5f6b9cc9a76df10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:46:40.792517 24744 ts_tablet_manager.cc:1905] T cf64c285a67043c1962226ed7a391aa4 P 00817077dc2d4558a5f6b9cc9a76df10: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 18:46:40 dist-test-slave-k6lg krb5kdc[24199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.129@KRBTEST.COM
I20260620 18:46:40.798532 24745 ts_tablet_manager.cc:1918] T cf64c285a67043c1962226ed7a391aa4 P 7c58463abab84e58b4772d01294a0c78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:46:40.798748 24745 log.cc:1196] T cf64c285a67043c1962226ed7a391aa4 P 7c58463abab84e58b4772d01294a0c78: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-0/wal/wals/cf64c285a67043c1962226ed7a391aa4
I20260620 18:46:40.799747 24745 ts_tablet_manager.cc:1939] T cf64c285a67043c1962226ed7a391aa4 P 7c58463abab84e58b4772d01294a0c78: Deleting consensus metadata
I20260620 18:46:40.800541 24744 ts_tablet_manager.cc:1918] T cf64c285a67043c1962226ed7a391aa4 P 00817077dc2d4558a5f6b9cc9a76df10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:46:40.800776 24744 log.cc:1196] T cf64c285a67043c1962226ed7a391aa4 P 00817077dc2d4558a5f6b9cc9a76df10: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980894474392-6022-0/minicluster-data/ts-1/wal/wals/cf64c285a67043c1962226ed7a391aa4
I20260620 18:46:40.801399 24744 ts_tablet_manager.cc:1939] T cf64c285a67043c1962226ed7a391aa4 P 00817077dc2d4558a5f6b9cc9a76df10: Deleting consensus metadata
I20260620 18:46:40.801905 24229 catalog_manager.cc:4841] TS 7c58463abab84e58b4772d01294a0c78 (127.5.225.129:35499): tablet cf64c285a67043c1962226ed7a391aa4 (table test-table [id=294b4004d2754bee825b50439efabf5b]) successfully deleted
I20260620 18:46:40.803246 24228 catalog_manager.cc:4841] TS 00817077dc2d4558a5f6b9cc9a76df10 (127.5.225.130:40113): tablet cf64c285a67043c1962226ed7a391aa4 (table test-table [id=294b4004d2754bee825b50439efabf5b]) successfully deleted
I20260620 18:46:40.805884 24716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:40.772286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52464 (local address 127.5.225.129:35499)
0620 18:46:40.772462 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:40.772468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:40.785584 (+ 13116us) server_negotiation.cc:407] Connection header received
0620 18:46:40.785801 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:40.785809 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:40.785913 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:40.786056 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:40.788098 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:40.789283 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:40.791727 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:40.792408 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:40.792512 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:40.798480 (+ 5968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:40.798497 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:40.798519 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:46:40.798569 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:40.802748 (+ 4179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:40.803739 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:40.803745 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:40.803747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:40.803872 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:40.804291 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:40.804294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:40.804296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:40.804619 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:40.804946 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:40.805243 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 18:46:40.805491 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:46:40.806627 24428 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:46:40.898063 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 24305
I20260620 18:46:40.917454 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 24439
Jun 20 18:46:40 dist-test-slave-k6lg krb5kdc[24199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.130@KRBTEST.COM
I20260620 18:46:40.935623 24717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:40.918434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46154 (local address 127.5.225.130:40113)
0620 18:46:40.918646 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:40.918653 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:40.918676 (+ 23us) server_negotiation.cc:407] Connection header received
0620 18:46:40.918748 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:40.918753 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:40.918842 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:40.918957 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:40.920326 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:40.921572 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:40.922929 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:40.923604 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:40.923729 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:40.928616 (+ 4887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:40.928636 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:40.928658 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:46:40.928703 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:40.932717 (+ 4014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:40.933590 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:40.933597 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:40.933599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:40.933740 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:40.934146 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:40.934149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:40.934152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:40.934589 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:40.934832 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:40.935172 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 18:46:40.935343 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:46:40.936528 24562 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:46:41.050002 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 24439
I20260620 18:46:41.068886 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 24572
Jun 20 18:46:41 dist-test-slave-k6lg krb5kdc[24199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.131@KRBTEST.COM
I20260620 18:46:41.084339 24714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:41.069714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47678 (local address 127.5.225.131:45873)
0620 18:46:41.069976 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:41.069984 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:41.070012 (+ 28us) server_negotiation.cc:407] Connection header received
0620 18:46:41.070087 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:41.070095 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:41.070205 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:41.070314 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:41.071617 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:41.072358 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:41.073597 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:41.073983 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:41.074072 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:41.078805 (+ 4733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:41.078819 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:41.078835 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:46:41.078876 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:41.081634 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:41.082366 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:41.082379 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:41.082385 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:41.082561 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:41.082900 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:41.082903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:41.082905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:41.083272 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:41.083480 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:41.083799 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 18:46:41.083971 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 18:46:41.085054 24695 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:46:41.173453 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 24572
I20260620 18:46:41.190307 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 24212
I20260620 18:46:41.191431 24274 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:46:41.303463 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 24212
2026-06-20T18:46:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4254 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T18:46:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:46:41Z Disabled control of system clock
I20260620 18:46:41.377779 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:46101
--webserver_interface=127.5.225.190
--webserver_port=0
--builtin_ntp_servers=127.5.225.148:37463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:46101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:46:41.705883 24754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:46:41.706194 24754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:46:41.706327 24754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:46:41.713524 24754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:46:41.713629 24754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:46:41.713666 24754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:46:41.713716 24754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:46:41.713759 24754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:46:41.724294 24754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:37463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:46101
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:46101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:46:41.726130 24754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:46:41.728343 24754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:46:41.740010 24760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:41.739943 24759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:41.740471 24762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:46:41.741012 24754 server_base.cc:1034] running on GCE node
I20260620 18:46:41.741779 24754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:46:41.743212 24754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:46:41.744405 24754 hybrid_clock.cc:648] HybridClock initialized: now 1781981201744346 us; error 58 us; skew 500 ppm
I20260620 18:46:41.744701 24754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:46:41.750918 24754 webserver.cc:460] Webserver started at http://127.5.225.190:45083/ using document root <none> and password file <none>
I20260620 18:46:41.751601 24754 fs_manager.cc:359] Metadata directory not provided
I20260620 18:46:41.751688 24754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:46:41.751940 24754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:46:41.754766 24754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "4cdc78914544465ea6fbadf99e5ccdbb"
format_stamp: "Formatted at 2026-06-20 18:46:41 on dist-test-slave-k6lg"
server_key: "a05b7813bcc7968918e42c978b7b780c"
server_key_iv: "3987cad1fe161db88476a6a0245fb001"
server_key_version: "encryptionkey@0"
I20260620 18:46:41.755513 24754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "4cdc78914544465ea6fbadf99e5ccdbb"
format_stamp: "Formatted at 2026-06-20 18:46:41 on dist-test-slave-k6lg"
server_key: "a05b7813bcc7968918e42c978b7b780c"
server_key_iv: "3987cad1fe161db88476a6a0245fb001"
server_key_version: "encryptionkey@0"
I20260620 18:46:41.760568 24754 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260620 18:46:41.764019 24768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:41.765401 24754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:46:41.765602 24754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "4cdc78914544465ea6fbadf99e5ccdbb"
format_stamp: "Formatted at 2026-06-20 18:46:41 on dist-test-slave-k6lg"
server_key: "a05b7813bcc7968918e42c978b7b780c"
server_key_iv: "3987cad1fe161db88476a6a0245fb001"
server_key_version: "encryptionkey@0"
I20260620 18:46:41.765838 24754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:46:41.795413 24754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:46:41.796371 24754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:46:41.796710 24754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:46:41.817123 24754 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:46101
I20260620 18:46:41.817145 24819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:46101 every 8 connection(s)
I20260620 18:46:41.818564 24754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:46:41.819025 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 24754
I20260620 18:46:41.819211 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:46:41.819815 6022 external_mini_cluster.cc:1428] Setting key 8a71523996edbca332ce06bda1515226
I20260620 18:46:41.823267 24820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:46:41.833066 24823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:41.821698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60758 (local address 127.5.225.190:46101)
0620 18:46:41.822772 (+ 1074us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:41.822783 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:41.822823 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:46:41.824552 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:41.824574 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:41.825010 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:41.825381 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:41.825965 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:41.827339 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:41.828686 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:41.829338 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:41.829532 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:41.830813 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:41.830820 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:41.830833 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:46:41.831378 (+ 545us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:41.831449 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:46:41.831472 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:41.831617 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:41.831839 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 18:46:41.832146 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":455,"threads_started":1}
I20260620 18:46:41.834194 24820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cdc78914544465ea6fbadf99e5ccdbb: Bootstrap starting.
I20260620 18:46:41.837474 24820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cdc78914544465ea6fbadf99e5ccdbb: Neither blocks nor log segments found. Creating new log.
I20260620 18:46:41.838644 24820 log.cc:824] T 00000000000000000000000000000000 P 4cdc78914544465ea6fbadf99e5ccdbb: Log is configured to *not* fsync() on all Append() calls
I20260620 18:46:41.841496 24820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cdc78914544465ea6fbadf99e5ccdbb: No bootstrap required, opened a new log
I20260620 18:46:41.848941 24820 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cdc78914544465ea6fbadf99e5ccdbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cdc78914544465ea6fbadf99e5ccdbb" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 46101 } }
I20260620 18:46:41.849308 24820 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4cdc78914544465ea6fbadf99e5ccdbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:46:41.849381 24820 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cdc78914544465ea6fbadf99e5ccdbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cdc78914544465ea6fbadf99e5ccdbb, State: Initialized, Role: FOLLOWER
I20260620 18:46:41.849874 24820 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cdc78914544465ea6fbadf99e5ccdbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cdc78914544465ea6fbadf99e5ccdbb" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 46101 } }
I20260620 18:46:41.850104 24820 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4cdc78914544465ea6fbadf99e5ccdbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:46:41.850373 24820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4cdc78914544465ea6fbadf99e5ccdbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:46:41.850560 24820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cdc78914544465ea6fbadf99e5ccdbb [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:46:41.853029 24820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4cdc78914544465ea6fbadf99e5ccdbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cdc78914544465ea6fbadf99e5ccdbb" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 46101 } }
I20260620 18:46:41.853502 24820 leader_election.cc:302] T 00000000000000000000000000000000 P 4cdc78914544465ea6fbadf99e5ccdbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cdc78914544465ea6fbadf99e5ccdbb; no voters:
I20260620 18:46:41.854046 24820 leader_election.cc:288] T 00000000000000000000000000000000 P 4cdc78914544465ea6fbadf99e5ccdbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:46:41.854283 24825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4cdc78914544465ea6fbadf99e5ccdbb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:46:41.855077 24825 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4cdc78914544465ea6fbadf99e5ccdbb [term 1 LEADER]: Becoming Leader. State: Replica: 4cdc78914544465ea6fbadf99e5ccdbb, State: Running, Role: LEADER
I20260620 18:46:41.855585 24825 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cdc78914544465ea6fbadf99e5ccdbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cdc78914544465ea6fbadf99e5ccdbb" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 46101 } }
I20260620 18:46:41.856248 24820 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cdc78914544465ea6fbadf99e5ccdbb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:46:41.858654 24827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cdc78914544465ea6fbadf99e5ccdbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cdc78914544465ea6fbadf99e5ccdbb. Latest consensus state: current_term: 1 leader_uuid: "4cdc78914544465ea6fbadf99e5ccdbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cdc78914544465ea6fbadf99e5ccdbb" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 46101 } } }
I20260620 18:46:41.858832 24826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cdc78914544465ea6fbadf99e5ccdbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cdc78914544465ea6fbadf99e5ccdbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cdc78914544465ea6fbadf99e5ccdbb" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 46101 } } }
I20260620 18:46:41.859089 24827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cdc78914544465ea6fbadf99e5ccdbb [sys.catalog]: This master's current role is: LEADER
I20260620 18:46:41.859097 24826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cdc78914544465ea6fbadf99e5ccdbb [sys.catalog]: This master's current role is: LEADER
I20260620 18:46:41.862098 24834 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:46:41.866582 24834 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:46:41.874903 24834 catalog_manager.cc:1347] Generated new cluster ID: 5ffd183d8a154cbf8669ed5b3af7a79d
I20260620 18:46:41.875015 24834 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:46:41.890139 24834 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:46:41.891335 24834 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:46:41.902714 24834 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4cdc78914544465ea6fbadf99e5ccdbb: Generated new TSK 0
I20260620 18:46:41.903721 24834 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:46:41.909246 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.129:0
--local_ip_for_outbound_sockets=127.5.225.129
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:46101
--builtin_ntp_servers=127.5.225.148:37463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:46:42.240121 24844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:46:42.240378 24844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:46:42.240463 24844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:46:42.247720 24844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:46:42.247812 24844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:46:42.247915 24844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.129
I20260620 18:46:42.256804 24844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:37463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:46101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.129
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:46:42.258095 24844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:46:42.260023 24844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:46:42.273335 24849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:42.273298 24850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:42.273250 24852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:46:42.274055 24844 server_base.cc:1034] running on GCE node
I20260620 18:46:42.274667 24844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:46:42.275678 24844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:46:42.276908 24844 hybrid_clock.cc:648] HybridClock initialized: now 1781981202276855 us; error 44 us; skew 500 ppm
I20260620 18:46:42.277254 24844 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:46:42.279578 24844 webserver.cc:460] Webserver started at http://127.5.225.129:41823/ using document root <none> and password file <none>
I20260620 18:46:42.280279 24844 fs_manager.cc:359] Metadata directory not provided
I20260620 18:46:42.280411 24844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:46:42.280733 24844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:46:42.283624 24844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-0/data/instance:
uuid: "617a7f264d364698a3b05b588c7ed2b8"
format_stamp: "Formatted at 2026-06-20 18:46:42 on dist-test-slave-k6lg"
server_key: "eee381efd22a6ced970f2e15a552438f"
server_key_iv: "427cf5119225befe25c2a69d29e2f4f5"
server_key_version: "encryptionkey@0"
I20260620 18:46:42.284420 24844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance:
uuid: "617a7f264d364698a3b05b588c7ed2b8"
format_stamp: "Formatted at 2026-06-20 18:46:42 on dist-test-slave-k6lg"
server_key: "eee381efd22a6ced970f2e15a552438f"
server_key_iv: "427cf5119225befe25c2a69d29e2f4f5"
server_key_version: "encryptionkey@0"
I20260620 18:46:42.289551 24844 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:46:42.293059 24858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:42.294452 24844 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:46:42.294627 24844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-0/wal
uuid: "617a7f264d364698a3b05b588c7ed2b8"
format_stamp: "Formatted at 2026-06-20 18:46:42 on dist-test-slave-k6lg"
server_key: "eee381efd22a6ced970f2e15a552438f"
server_key_iv: "427cf5119225befe25c2a69d29e2f4f5"
server_key_version: "encryptionkey@0"
I20260620 18:46:42.294852 24844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:46:42.312745 24844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:46:42.313592 24844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:46:42.313898 24844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:46:42.314934 24844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:46:42.316358 24844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:46:42.316483 24844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:42.316609 24844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:46:42.316696 24844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:42.349711 24844 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.129:45531
I20260620 18:46:42.349799 24970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.129:45531 every 8 connection(s)
I20260620 18:46:42.351169 24844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
I20260620 18:46:42.352665 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 24844
I20260620 18:46:42.352835 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance
I20260620 18:46:42.353353 6022 external_mini_cluster.cc:1428] Setting key c4c9abc5f80046c7bd25043f8f7869a5
I20260620 18:46:42.356622 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.130:0
--local_ip_for_outbound_sockets=127.5.225.130
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:46101
--builtin_ntp_servers=127.5.225.148:37463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:46:42.367027 24973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:42.353876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:35027 (local address 127.5.225.190:46101)
0620 18:46:42.354468 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:42.354475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:42.356097 (+ 1622us) server_negotiation.cc:407] Connection header received
0620 18:46:42.359011 (+ 2914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:42.359015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:42.359090 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:42.359223 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:42.360605 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:42.361350 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:42.363932 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:42.364323 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:42.364424 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:42.366183 (+ 1759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:42.366192 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:42.366196 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:46:42.366222 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:42.366284 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:46:42.366304 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:42.366415 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:42.366687 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 18:46:42.366773 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":352,"threads_started":1}
I20260620 18:46:42.368134 24974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:42.354595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:46101 (local address 127.5.225.129:35027)
0620 18:46:42.355925 (+ 1330us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:42.355993 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:42.358683 (+ 2690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:42.359491 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:42.359506 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:42.359627 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:46:42.360427 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:42.360455 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:42.361519 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:42.361527 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:42.363782 (+ 2255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:42.363799 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:42.364554 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:42.364564 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:42.364902 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:42.365784 (+ 882us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:46:42.365786 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:46:42.365828 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:46:42.365916 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:46:42.366449 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:46:42.366459 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:46:42.366466 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 18:46:42.366640 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 18:46:42.367007 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"thread_start_us":563,"threads_started":1}
I20260620 18:46:42.369851 24971 heartbeater.cc:347] Connected to a master server at 127.5.225.190:46101
I20260620 18:46:42.370271 24971 heartbeater.cc:464] Registering TS with master...
I20260620 18:46:42.371096 24971 heartbeater.cc:511] Master 127.5.225.190:46101 requested a full tablet report, sending...
I20260620 18:46:42.373297 24785 ts_manager.cc:194] Registered new tserver with Master: 617a7f264d364698a3b05b588c7ed2b8 (127.5.225.129:45531)
I20260620 18:46:42.375031 24785 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.225.129:35027
W20260620 18:46:42.685993 24975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:46:42.686347 24975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:46:42.686517 24975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:46:42.693588 24975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:46:42.693674 24975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:46:42.693883 24975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.130
I20260620 18:46:42.703013 24975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:37463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:46101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.130
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:46:42.704284 24975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:46:42.706037 24975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:46:42.719188 24981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:42.719360 24980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:42.719368 24983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:46:42.720464 24975 server_base.cc:1034] running on GCE node
I20260620 18:46:42.721130 24975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:46:42.722064 24975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:46:42.723305 24975 hybrid_clock.cc:648] HybridClock initialized: now 1781981202723256 us; error 36 us; skew 500 ppm
I20260620 18:46:42.723655 24975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:46:42.725862 24975 webserver.cc:460] Webserver started at http://127.5.225.130:46777/ using document root <none> and password file <none>
I20260620 18:46:42.726600 24975 fs_manager.cc:359] Metadata directory not provided
I20260620 18:46:42.726716 24975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:46:42.727044 24975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:46:42.729846 24975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-1/data/instance:
uuid: "567bacc61bc0415586646dba2169b956"
format_stamp: "Formatted at 2026-06-20 18:46:42 on dist-test-slave-k6lg"
server_key: "efe4ce065875b8042751892745a21f1d"
server_key_iv: "8bc1ead452a44eb1792ad6d644a6975c"
server_key_version: "encryptionkey@0"
I20260620 18:46:42.730692 24975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance:
uuid: "567bacc61bc0415586646dba2169b956"
format_stamp: "Formatted at 2026-06-20 18:46:42 on dist-test-slave-k6lg"
server_key: "efe4ce065875b8042751892745a21f1d"
server_key_iv: "8bc1ead452a44eb1792ad6d644a6975c"
server_key_version: "encryptionkey@0"
I20260620 18:46:42.735723 24975 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:46:42.739431 24989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:42.740756 24975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:46:42.740932 24975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-1/wal
uuid: "567bacc61bc0415586646dba2169b956"
format_stamp: "Formatted at 2026-06-20 18:46:42 on dist-test-slave-k6lg"
server_key: "efe4ce065875b8042751892745a21f1d"
server_key_iv: "8bc1ead452a44eb1792ad6d644a6975c"
server_key_version: "encryptionkey@0"
I20260620 18:46:42.741163 24975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:46:42.766294 24975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:46:42.767234 24975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:46:42.767580 24975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:46:42.768692 24975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:46:42.770525 24975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:46:42.770639 24975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:42.770777 24975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:46:42.770889 24975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:42.803458 24975 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.130:39991
I20260620 18:46:42.803551 25101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.130:39991 every 8 connection(s)
I20260620 18:46:42.804776 24975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
I20260620 18:46:42.815066 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 24975
I20260620 18:46:42.815243 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance
I20260620 18:46:42.815717 6022 external_mini_cluster.cc:1428] Setting key c5cee42c725f922e0d7ba30d6f883537
I20260620 18:46:42.817255 24973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:42.807379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:50851 (local address 127.5.225.190:46101)
0620 18:46:42.807611 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:42.807618 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:42.808920 (+ 1302us) server_negotiation.cc:407] Connection header received
0620 18:46:42.810748 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:42.810753 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:42.810821 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:42.810913 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:42.811917 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:42.812637 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:42.814097 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:42.814564 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:42.814695 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:42.816084 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:42.816091 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:42.816097 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:46:42.816119 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:42.816180 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:46:42.816198 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:42.816305 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:42.816840 (+ 535us) server_negotiation.cc:299] Negotiation successful
0620 18:46:42.816939 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 18:46:42.818501 25104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:42.807901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:46101 (local address 127.5.225.130:50851)
0620 18:46:42.808760 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:42.808800 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:42.810480 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:42.811106 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:42.811113 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:42.811214 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:46:42.811759 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:42.811774 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:42.812769 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:42.812772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:42.813962 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:42.813973 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:42.814766 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:42.814771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:42.814963 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:42.815727 (+ 764us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:46:42.815730 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:46:42.815769 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:46:42.815884 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:46:42.816597 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:46:42.816607 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:46:42.816615 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 18:46:42.816802 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 18:46:42.817210 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":257,"threads_started":1}
I20260620 18:46:42.819033 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.131:0
--local_ip_for_outbound_sockets=127.5.225.131
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:46101
--builtin_ntp_servers=127.5.225.148:37463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:46:42.820024 25102 heartbeater.cc:347] Connected to a master server at 127.5.225.190:46101
I20260620 18:46:42.820338 25102 heartbeater.cc:464] Registering TS with master...
I20260620 18:46:42.821203 25102 heartbeater.cc:511] Master 127.5.225.190:46101 requested a full tablet report, sending...
I20260620 18:46:42.822770 24785 ts_manager.cc:194] Registered new tserver with Master: 567bacc61bc0415586646dba2169b956 (127.5.225.130:39991)
I20260620 18:46:42.823843 24785 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.225.130:50851
W20260620 18:46:43.144711 25105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:46:43.145048 25105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:46:43.145202 25105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:46:43.152415 25105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:46:43.152503 25105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:46:43.152675 25105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.131
I20260620 18:46:43.161648 25105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:37463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:46101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.131
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:46:43.162957 25105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:46:43.164723 25105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:46:43.177697 25110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:43.177671 25113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:43.177729 25111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:46:43.178653 25105 server_base.cc:1034] running on GCE node
I20260620 18:46:43.179414 25105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:46:43.180406 25105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:46:43.181644 25105 hybrid_clock.cc:648] HybridClock initialized: now 1781981203181590 us; error 40 us; skew 500 ppm
I20260620 18:46:43.182018 25105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:46:43.184609 25105 webserver.cc:460] Webserver started at http://127.5.225.131:42353/ using document root <none> and password file <none>
I20260620 18:46:43.185392 25105 fs_manager.cc:359] Metadata directory not provided
I20260620 18:46:43.185513 25105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:46:43.185882 25105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:46:43.188887 25105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-2/data/instance:
uuid: "76c804b22c6b4a80aa71ee99553c918b"
format_stamp: "Formatted at 2026-06-20 18:46:43 on dist-test-slave-k6lg"
server_key: "6b27575b5df3f58d399001d082ec6602"
server_key_iv: "c929555675ec70726c13f53922da0538"
server_key_version: "encryptionkey@0"
I20260620 18:46:43.189800 25105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance:
uuid: "76c804b22c6b4a80aa71ee99553c918b"
format_stamp: "Formatted at 2026-06-20 18:46:43 on dist-test-slave-k6lg"
server_key: "6b27575b5df3f58d399001d082ec6602"
server_key_iv: "c929555675ec70726c13f53922da0538"
server_key_version: "encryptionkey@0"
I20260620 18:46:43.195053 25105 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:46:43.198726 25119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:43.200191 25105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 18:46:43.200379 25105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-2/wal
uuid: "76c804b22c6b4a80aa71ee99553c918b"
format_stamp: "Formatted at 2026-06-20 18:46:43 on dist-test-slave-k6lg"
server_key: "6b27575b5df3f58d399001d082ec6602"
server_key_iv: "c929555675ec70726c13f53922da0538"
server_key_version: "encryptionkey@0"
I20260620 18:46:43.200618 25105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:46:43.215725 25105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:46:43.216637 25105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:46:43.216972 25105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:46:43.218055 25105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:46:43.219687 25105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:46:43.219830 25105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:43.219975 25105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:46:43.220065 25105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:43.256044 25105 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.131:35701
I20260620 18:46:43.256090 25231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.131:35701 every 8 connection(s)
I20260620 18:46:43.257550 25105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
I20260620 18:46:43.265283 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 25105
I20260620 18:46:43.265496 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance
I20260620 18:46:43.266007 6022 external_mini_cluster.cc:1428] Setting key 410d7d7177d9dfa713ba2bfaa8c64c28
I20260620 18:46:43.271896 24973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:43.260481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:43525 (local address 127.5.225.190:46101)
0620 18:46:43.260700 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:43.260706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:43.262243 (+ 1537us) server_negotiation.cc:407] Connection header received
0620 18:46:43.264165 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:43.264171 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:43.264273 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:43.264401 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:43.265641 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:43.266496 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:43.268515 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:43.269166 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:43.269340 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:43.271014 (+ 1674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:43.271020 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:43.271025 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:46:43.271054 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:43.271088 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:46:43.271101 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:43.271196 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:43.271450 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 18:46:43.271557 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:46:43.273066 25234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:43.260986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:46101 (local address 127.5.225.131:43525)
0620 18:46:43.262025 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:43.262067 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:43.263846 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:43.264653 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:43.264664 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:43.264767 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:46:43.265416 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:43.265462 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:43.266668 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:43.266674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:43.268362 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:43.268375 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:43.269440 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:43.269448 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:43.269712 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:43.270607 (+ 895us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:46:43.270610 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:46:43.270654 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:46:43.270744 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:46:43.271211 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:46:43.271225 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:46:43.271237 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 18:46:43.271434 (+ 197us) client_negotiation.cc:240] Negotiation successful
0620 18:46:43.271914 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":318,"threads_started":1}
I20260620 18:46:43.274885 25232 heartbeater.cc:347] Connected to a master server at 127.5.225.190:46101
I20260620 18:46:43.275298 25232 heartbeater.cc:464] Registering TS with master...
I20260620 18:46:43.276172 25232 heartbeater.cc:511] Master 127.5.225.190:46101 requested a full tablet report, sending...
I20260620 18:46:43.277845 24785 ts_manager.cc:194] Registered new tserver with Master: 76c804b22c6b4a80aa71ee99553c918b (127.5.225.131:35701)
I20260620 18:46:43.278834 24785 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.225.131:43525
I20260620 18:46:43.281607 6022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:46:43.292851 24973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:43.286554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60766 (local address 127.5.225.190:46101)
0620 18:46:43.286787 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:43.286798 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:43.287178 (+ 380us) server_negotiation.cc:407] Connection header received
0620 18:46:43.287323 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:43.287327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:43.287403 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:43.287489 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:43.287892 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:43.288590 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:43.290149 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:43.290687 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:43.290805 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:43.291998 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:43.292003 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:43.292006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:46:43.292026 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:43.292072 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:46:43.292085 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:43.292241 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:43.292410 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 18:46:43.292503 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:46:43.296980 24785 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:46:43.304708 24785 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:46:43.329131 25244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:43.317552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:45531 (local address 127.0.0.1:50676)
0620 18:46:43.318792 (+ 1240us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:43.318858 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:43.319060 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:43.320366 (+ 1306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:43.320375 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:43.320410 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:46:43.320829 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:43.320843 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:43.323092 (+ 2249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:43.323098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:43.325847 (+ 2749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:43.325863 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:43.328052 (+ 2189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:43.328067 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:43.328306 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:43.328356 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 18:46:43.328474 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 18:46:43.328683 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"mutex_wait_us":3,"thread_start_us":571,"threads_started":1}
I20260620 18:46:43.329310 25246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:43.319062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.130:39991 (local address 127.0.0.1:57202)
0620 18:46:43.319676 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:43.319696 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:43.319821 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:43.320987 (+ 1166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:43.320993 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:43.321023 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:46:43.321436 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:43.321449 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:43.323728 (+ 2279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:43.323734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:43.326135 (+ 2401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:43.326148 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:43.328080 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:43.328087 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:43.328690 (+ 603us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:43.328722 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 18:46:43.328811 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 18:46:43.328934 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":391,"thread_start_us":298,"threads_started":1}
I20260620 18:46:43.330384 25247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:43.319665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57202 (local address 127.5.225.130:39991)
0620 18:46:43.320360 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:43.320369 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:43.320395 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:46:43.320504 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:43.320513 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:43.320766 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:43.321661 (+ 895us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:46:43.321880 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:43.323555 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:43.326397 (+ 2842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:43.327864 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:43.328157 (+ 293us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:43.328421 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:43.329765 (+ 1344us) server_negotiation.cc:299] Negotiation successful
0620 18:46:43.329990 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":413,"threads_started":1}
I20260620 18:46:43.330821 25245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:43.317795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50676 (local address 127.5.225.129:45531)
0620 18:46:43.319660 (+ 1865us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:43.319670 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:43.319701 (+ 31us) server_negotiation.cc:407] Connection header received
0620 18:46:43.319800 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:43.319810 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:43.320151 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:43.320347 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:46:43.321065 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:43.322895 (+ 1830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:43.326145 (+ 3250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:43.327789 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:43.329701 (+ 1912us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:43.329984 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:43.330065 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 18:46:43.330291 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1285,"thread_start_us":1167,"threads_started":1}
I20260620 18:46:43.335357 25037 tablet_service.cc:1467] Processing CreateTablet for tablet d08ae625f9514546a853913ae1564b6c (DEFAULT_TABLE table=test-table [id=d484d00cd19147699e7198ec80a6e98a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:46:43.335357 24906 tablet_service.cc:1467] Processing CreateTablet for tablet d08ae625f9514546a853913ae1564b6c (DEFAULT_TABLE table=test-table [id=d484d00cd19147699e7198ec80a6e98a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:46:43.336881 24906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d08ae625f9514546a853913ae1564b6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:46:43.336880 25037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d08ae625f9514546a853913ae1564b6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:46:43.337646 25244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:43.319727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.131:35701 (local address 127.0.0.1:42192)
0620 18:46:43.329432 (+ 9705us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:43.329454 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:43.329574 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:43.331963 (+ 2389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:43.331968 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:43.331994 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:46:43.332157 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:43.332166 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:43.333668 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:43.333674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:43.335167 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:43.335182 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:43.336805 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:43.336812 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:43.337019 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:43.337063 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 18:46:43.337165 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 18:46:43.337326 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9590,"thread_start_us":11486,"threads_started":1}
I20260620 18:46:43.338052 25249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:43.320266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42192 (local address 127.5.225.131:35701)
0620 18:46:43.331279 (+ 11013us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:43.331291 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:43.331333 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:46:43.331461 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:43.331469 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:43.331730 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:43.331960 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:46:43.332333 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:43.333511 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:43.335367 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:43.336622 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:43.336779 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:43.337076 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:43.337280 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 18:46:43.337446 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10706,"thread_start_us":10547,"threads_started":1}
I20260620 18:46:43.341717 25167 tablet_service.cc:1467] Processing CreateTablet for tablet d08ae625f9514546a853913ae1564b6c (DEFAULT_TABLE table=test-table [id=d484d00cd19147699e7198ec80a6e98a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:46:43.343219 25167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d08ae625f9514546a853913ae1564b6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:46:43.358089 25250 tablet_bootstrap.cc:492] T d08ae625f9514546a853913ae1564b6c P 567bacc61bc0415586646dba2169b956: Bootstrap starting.
I20260620 18:46:43.362931 25251 tablet_bootstrap.cc:492] T d08ae625f9514546a853913ae1564b6c P 617a7f264d364698a3b05b588c7ed2b8: Bootstrap starting.
I20260620 18:46:43.363309 25252 tablet_bootstrap.cc:492] T d08ae625f9514546a853913ae1564b6c P 76c804b22c6b4a80aa71ee99553c918b: Bootstrap starting.
I20260620 18:46:43.365046 25250 tablet_bootstrap.cc:654] T d08ae625f9514546a853913ae1564b6c P 567bacc61bc0415586646dba2169b956: Neither blocks nor log segments found. Creating new log.
I20260620 18:46:43.366214 25250 log.cc:824] T d08ae625f9514546a853913ae1564b6c P 567bacc61bc0415586646dba2169b956: Log is configured to *not* fsync() on all Append() calls
I20260620 18:46:43.366864 25252 tablet_bootstrap.cc:654] T d08ae625f9514546a853913ae1564b6c P 76c804b22c6b4a80aa71ee99553c918b: Neither blocks nor log segments found. Creating new log.
I20260620 18:46:43.366864 25251 tablet_bootstrap.cc:654] T d08ae625f9514546a853913ae1564b6c P 617a7f264d364698a3b05b588c7ed2b8: Neither blocks nor log segments found. Creating new log.
I20260620 18:46:43.368037 25252 log.cc:824] T d08ae625f9514546a853913ae1564b6c P 76c804b22c6b4a80aa71ee99553c918b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:46:43.368088 25251 log.cc:824] T d08ae625f9514546a853913ae1564b6c P 617a7f264d364698a3b05b588c7ed2b8: Log is configured to *not* fsync() on all Append() calls
I20260620 18:46:43.370091 25250 tablet_bootstrap.cc:492] T d08ae625f9514546a853913ae1564b6c P 567bacc61bc0415586646dba2169b956: No bootstrap required, opened a new log
I20260620 18:46:43.370428 25250 ts_tablet_manager.cc:1397] T d08ae625f9514546a853913ae1564b6c P 567bacc61bc0415586646dba2169b956: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.002s
I20260620 18:46:43.373016 25252 tablet_bootstrap.cc:492] T d08ae625f9514546a853913ae1564b6c P 76c804b22c6b4a80aa71ee99553c918b: No bootstrap required, opened a new log
I20260620 18:46:43.373279 25252 ts_tablet_manager.cc:1397] T d08ae625f9514546a853913ae1564b6c P 76c804b22c6b4a80aa71ee99553c918b: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260620 18:46:43.373546 25251 tablet_bootstrap.cc:492] T d08ae625f9514546a853913ae1564b6c P 617a7f264d364698a3b05b588c7ed2b8: No bootstrap required, opened a new log
I20260620 18:46:43.373817 25251 ts_tablet_manager.cc:1397] T d08ae625f9514546a853913ae1564b6c P 617a7f264d364698a3b05b588c7ed2b8: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260620 18:46:43.378293 24971 heartbeater.cc:503] Master 127.5.225.190:46101 was elected leader, sending a full tablet report...
I20260620 18:46:43.380971 25252 raft_consensus.cc:348] T d08ae625f9514546a853913ae1564b6c P 76c804b22c6b4a80aa71ee99553c918b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617a7f264d364698a3b05b588c7ed2b8" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 45531 } } peers { permanent_uuid: "567bacc61bc0415586646dba2169b956" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39991 } } peers { permanent_uuid: "76c804b22c6b4a80aa71ee99553c918b" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 35701 } }
I20260620 18:46:43.381428 25252 raft_consensus.cc:374] T d08ae625f9514546a853913ae1564b6c P 76c804b22c6b4a80aa71ee99553c918b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:46:43.381527 25252 raft_consensus.cc:729] T d08ae625f9514546a853913ae1564b6c P 76c804b22c6b4a80aa71ee99553c918b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c804b22c6b4a80aa71ee99553c918b, State: Initialized, Role: FOLLOWER
I20260620 18:46:43.382285 25252 consensus_queue.cc:260] T d08ae625f9514546a853913ae1564b6c P 76c804b22c6b4a80aa71ee99553c918b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617a7f264d364698a3b05b588c7ed2b8" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 45531 } } peers { permanent_uuid: "567bacc61bc0415586646dba2169b956" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39991 } } peers { permanent_uuid: "76c804b22c6b4a80aa71ee99553c918b" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 35701 } }
I20260620 18:46:43.382218 25250 raft_consensus.cc:348] T d08ae625f9514546a853913ae1564b6c P 567bacc61bc0415586646dba2169b956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617a7f264d364698a3b05b588c7ed2b8" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 45531 } } peers { permanent_uuid: "567bacc61bc0415586646dba2169b956" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39991 } } peers { permanent_uuid: "76c804b22c6b4a80aa71ee99553c918b" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 35701 } }
I20260620 18:46:43.382650 25250 raft_consensus.cc:374] T d08ae625f9514546a853913ae1564b6c P 567bacc61bc0415586646dba2169b956 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:46:43.382746 25250 raft_consensus.cc:729] T d08ae625f9514546a853913ae1564b6c P 567bacc61bc0415586646dba2169b956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 567bacc61bc0415586646dba2169b956, State: Initialized, Role: FOLLOWER
I20260620 18:46:43.383487 25250 consensus_queue.cc:260] T d08ae625f9514546a853913ae1564b6c P 567bacc61bc0415586646dba2169b956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617a7f264d364698a3b05b588c7ed2b8" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 45531 } } peers { permanent_uuid: "567bacc61bc0415586646dba2169b956" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39991 } } peers { permanent_uuid: "76c804b22c6b4a80aa71ee99553c918b" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 35701 } }
I20260620 18:46:43.385309 25102 heartbeater.cc:503] Master 127.5.225.190:46101 was elected leader, sending a full tablet report...
I20260620 18:46:43.385937 25250 ts_tablet_manager.cc:1428] T d08ae625f9514546a853913ae1564b6c P 567bacc61bc0415586646dba2169b956: Time spent starting tablet: real 0.015s user 0.007s sys 0.008s
I20260620 18:46:43.386245 25251 raft_consensus.cc:348] T d08ae625f9514546a853913ae1564b6c P 617a7f264d364698a3b05b588c7ed2b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617a7f264d364698a3b05b588c7ed2b8" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 45531 } } peers { permanent_uuid: "567bacc61bc0415586646dba2169b956" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39991 } } peers { permanent_uuid: "76c804b22c6b4a80aa71ee99553c918b" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 35701 } }
I20260620 18:46:43.386680 25251 raft_consensus.cc:374] T d08ae625f9514546a853913ae1564b6c P 617a7f264d364698a3b05b588c7ed2b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:46:43.386754 25232 heartbeater.cc:503] Master 127.5.225.190:46101 was elected leader, sending a full tablet report...
I20260620 18:46:43.386780 25251 raft_consensus.cc:729] T d08ae625f9514546a853913ae1564b6c P 617a7f264d364698a3b05b588c7ed2b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 617a7f264d364698a3b05b588c7ed2b8, State: Initialized, Role: FOLLOWER
I20260620 18:46:43.387413 25251 consensus_queue.cc:260] T d08ae625f9514546a853913ae1564b6c P 617a7f264d364698a3b05b588c7ed2b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617a7f264d364698a3b05b588c7ed2b8" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 45531 } } peers { permanent_uuid: "567bacc61bc0415586646dba2169b956" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39991 } } peers { permanent_uuid: "76c804b22c6b4a80aa71ee99553c918b" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 35701 } }
I20260620 18:46:43.388096 25252 ts_tablet_manager.cc:1428] T d08ae625f9514546a853913ae1564b6c P 76c804b22c6b4a80aa71ee99553c918b: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260620 18:46:43.390080 25251 ts_tablet_manager.cc:1428] T d08ae625f9514546a853913ae1564b6c P 617a7f264d364698a3b05b588c7ed2b8: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
W20260620 18:46:43.509699 25233 tablet.cc:2369] T d08ae625f9514546a853913ae1564b6c P 76c804b22c6b4a80aa71ee99553c918b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:46:43.556949 25103 tablet.cc:2369] T d08ae625f9514546a853913ae1564b6c P 567bacc61bc0415586646dba2169b956: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:46:43.595099 25257 raft_consensus.cc:482] T d08ae625f9514546a853913ae1564b6c P 567bacc61bc0415586646dba2169b956 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:46:43.595530 25257 raft_consensus.cc:504] T d08ae625f9514546a853913ae1564b6c P 567bacc61bc0415586646dba2169b956 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617a7f264d364698a3b05b588c7ed2b8" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 45531 } } peers { permanent_uuid: "567bacc61bc0415586646dba2169b956" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39991 } } peers { permanent_uuid: "76c804b22c6b4a80aa71ee99553c918b" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 35701 } }
I20260620 18:46:43.596961 25256 raft_consensus.cc:482] T d08ae625f9514546a853913ae1564b6c P 76c804b22c6b4a80aa71ee99553c918b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:46:43.597232 25256 raft_consensus.cc:504] T d08ae625f9514546a853913ae1564b6c P 76c804b22c6b4a80aa71ee99553c918b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617a7f264d364698a3b05b588c7ed2b8" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 45531 } } peers { permanent_uuid: "567bacc61bc0415586646dba2169b956" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39991 } } peers { permanent_uuid: "76c804b22c6b4a80aa71ee99553c918b" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 35701 } }
I20260620 18:46:43.597550 25257 leader_election.cc:288] T d08ae625f9514546a853913ae1564b6c P 567bacc61bc0415586646dba2169b956 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 617a7f264d364698a3b05b588c7ed2b8 (127.5.225.129:45531), 76c804b22c6b4a80aa71ee99553c918b (127.5.225.131:35701)
I20260620 18:46:43.599169 25256 leader_election.cc:288] T d08ae625f9514546a853913ae1564b6c P 76c804b22c6b4a80aa71ee99553c918b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 617a7f264d364698a3b05b588c7ed2b8 (127.5.225.129:45531), 567bacc61bc0415586646dba2169b956 (127.5.225.130:39991)
W20260620 18:46:43.603626 24972 tablet.cc:2369] T d08ae625f9514546a853913ae1564b6c P 617a7f264d364698a3b05b588c7ed2b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:46:43.608670 25245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:43.598023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:54335 (local address 127.5.225.129:45531)
0620 18:46:43.598281 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:43.598287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:43.598802 (+ 515us) server_negotiation.cc:407] Connection header received
0620 18:46:43.599111 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:43.599117 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:43.599237 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:43.599327 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:46:43.600301 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:43.601594 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:43.605182 (+ 3588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:43.607019 (+ 1837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:43.607183 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:43.607292 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:43.608047 (+ 755us) server_negotiation.cc:299] Negotiation successful
0620 18:46:43.608249 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:46:43.609695 25249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:43.599088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:49841 (local address 127.5.225.131:35701)
0620 18:46:43.599490 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:43.599498 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:43.599833 (+ 335us) server_negotiation.cc:407] Connection header received
0620 18:46:43.600123 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:43.600132 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:43.600230 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:43.600305 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:46:43.601299 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:43.602773 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:43.606005 (+ 3232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:43.607770 (+ 1765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:43.607938 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:43.608042 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:43.609088 (+ 1046us) server_negotiation.cc:299] Negotiation successful
0620 18:46:43.609281 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"spinlock_wait_cycles":27520}
I20260620 18:46:43.609953 25259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:43.597947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:45531 (local address 127.5.225.130:54335)
0620 18:46:43.598700 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:43.598722 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:43.598920 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:43.599610 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:43.599615 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:43.599653 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:46:43.600125 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:43.600140 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:43.601829 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:43.601837 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:43.604883 (+ 3046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:43.604904 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:43.607529 (+ 2625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:43.607538 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:43.607799 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:43.607853 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 18:46:43.609416 (+ 1563us) client_negotiation.cc:240] Negotiation successful
0620 18:46:43.609576 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":429,"threads_started":1}
I20260620 18:46:43.610949 25260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:43.598841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.131:35701 (local address 127.5.225.130:49841)
0620 18:46:43.599718 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:43.599737 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:43.599946 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:43.600607 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:43.600614 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:43.600647 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:46:43.601095 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:43.601109 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:43.603002 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:43.603010 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:43.605754 (+ 2744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:43.605775 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:43.608239 (+ 2464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:43.608248 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:43.608512 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:43.608563 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 18:46:43.610312 (+ 1749us) client_negotiation.cc:240] Negotiation successful
0620 18:46:43.610518 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":1094,"threads_started":1}
I20260620 18:46:43.612808 25187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d08ae625f9514546a853913ae1564b6c" candidate_uuid: "567bacc61bc0415586646dba2169b956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c804b22c6b4a80aa71ee99553c918b" is_pre_election: true
I20260620 18:46:43.613078 24926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d08ae625f9514546a853913ae1564b6c" candidate_uuid: "567bacc61bc0415586646dba2169b956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "617a7f264d364698a3b05b588c7ed2b8" is_pre_election: true
I20260620 18:46:43.613339 25187 raft_consensus.cc:2454] T d08ae625f9514546a853913ae1564b6c P 76c804b22c6b4a80aa71ee99553c918b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 567bacc61bc0415586646dba2169b956 in term 0.
I20260620 18:46:43.613801 24926 raft_consensus.cc:2454] T d08ae625f9514546a853913ae1564b6c P 617a7f264d364698a3b05b588c7ed2b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 567bacc61bc0415586646dba2169b956 in term 0.
I20260620 18:46:43.614120 24993 leader_election.cc:302] T d08ae625f9514546a853913ae1564b6c P 567bacc61bc0415586646dba2169b956 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 567bacc61bc0415586646dba2169b956, 76c804b22c6b4a80aa71ee99553c918b; no voters:
I20260620 18:46:43.614653 25257 raft_consensus.cc:2781] T d08ae625f9514546a853913ae1564b6c P 567bacc61bc0415586646dba2169b956 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:46:43.614830 25257 raft_consensus.cc:482] T d08ae625f9514546a853913ae1564b6c P 567bacc61bc0415586646dba2169b956 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:46:43.614934 25257 raft_consensus.cc:3037] T d08ae625f9514546a853913ae1564b6c P 567bacc61bc0415586646dba2169b956 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:46:43.616088 25234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:43.599547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:45531 (local address 127.5.225.131:51531)
0620 18:46:43.600251 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:43.600271 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:43.600364 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:43.609333 (+ 8969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:43.609339 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:43.609370 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:46:43.609578 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:43.609588 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:43.611382 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:43.611392 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:43.613559 (+ 2167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:43.613573 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:43.615319 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:43.615325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:43.615517 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:43.615551 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 18:46:43.615642 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 18:46:43.615778 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582}
I20260620 18:46:43.616322 25245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:43.599700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:51531 (local address 127.5.225.129:45531)
0620 18:46:43.608989 (+ 9289us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:43.608997 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:43.609021 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:46:43.609092 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:43.609099 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:43.609205 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:43.609311 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:46:43.609765 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:43.611179 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:43.613757 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:43.615012 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:43.615166 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:43.615229 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:43.615904 (+ 675us) server_negotiation.cc:299] Negotiation successful
0620 18:46:43.616030 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9134,"thread_start_us":12394,"threads_started":1}
I20260620 18:46:43.616881 24926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d08ae625f9514546a853913ae1564b6c" candidate_uuid: "76c804b22c6b4a80aa71ee99553c918b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "617a7f264d364698a3b05b588c7ed2b8" is_pre_election: true
I20260620 18:46:43.617100 24926 raft_consensus.cc:2454] T d08ae625f9514546a853913ae1564b6c P 617a7f264d364698a3b05b588c7ed2b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c804b22c6b4a80aa71ee99553c918b in term 0.
I20260620 18:46:43.617611 25123 leader_election.cc:302] T d08ae625f9514546a853913ae1564b6c P 76c804b22c6b4a80aa71ee99553c918b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 617a7f264d364698a3b05b588c7ed2b8, 76c804b22c6b4a80aa71ee99553c918b; no voters:
I20260620 18:46:43.618032 25256 raft_consensus.cc:2781] T d08ae625f9514546a853913ae1564b6c P 76c804b22c6b4a80aa71ee99553c918b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:46:43.618252 25256 raft_consensus.cc:482] T d08ae625f9514546a853913ae1564b6c P 76c804b22c6b4a80aa71ee99553c918b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:46:43.618382 25256 raft_consensus.cc:3037] T d08ae625f9514546a853913ae1564b6c P 76c804b22c6b4a80aa71ee99553c918b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:46:43.618217 25257 raft_consensus.cc:504] T d08ae625f9514546a853913ae1564b6c P 567bacc61bc0415586646dba2169b956 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617a7f264d364698a3b05b588c7ed2b8" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 45531 } } peers { permanent_uuid: "567bacc61bc0415586646dba2169b956" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39991 } } peers { permanent_uuid: "76c804b22c6b4a80aa71ee99553c918b" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 35701 } }
I20260620 18:46:43.618978 25257 leader_election.cc:288] T d08ae625f9514546a853913ae1564b6c P 567bacc61bc0415586646dba2169b956 [CANDIDATE]: Term 1 election: Requested vote from peers 617a7f264d364698a3b05b588c7ed2b8 (127.5.225.129:45531), 76c804b22c6b4a80aa71ee99553c918b (127.5.225.131:35701)
I20260620 18:46:43.619520 24926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d08ae625f9514546a853913ae1564b6c" candidate_uuid: "567bacc61bc0415586646dba2169b956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "617a7f264d364698a3b05b588c7ed2b8"
I20260620 18:46:43.619504 25187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d08ae625f9514546a853913ae1564b6c" candidate_uuid: "567bacc61bc0415586646dba2169b956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c804b22c6b4a80aa71ee99553c918b"
I20260620 18:46:43.619722 24926 raft_consensus.cc:3037] T d08ae625f9514546a853913ae1564b6c P 617a7f264d364698a3b05b588c7ed2b8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:46:43.620504 25247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:43.600615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:50403 (local address 127.5.225.130:39991)
0620 18:46:43.600793 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:43.600798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:43.612472 (+ 11674us) server_negotiation.cc:407] Connection header received
0620 18:46:43.612737 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:43.612743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:43.612831 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:43.612892 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:46:43.614525 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:43.615588 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:43.618124 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:43.619396 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:43.619526 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:43.619591 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:43.620145 (+ 554us) server_negotiation.cc:299] Negotiation successful
0620 18:46:43.620266 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 18:46:43.620594 25262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:43.599886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.130:39991 (local address 127.5.225.131:50403)
0620 18:46:43.612357 (+ 12471us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:43.612375 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:43.612578 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:43.613077 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:43.613084 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:43.613111 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:46:43.614300 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:43.614316 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:43.615778 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:43.615783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:43.617867 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:43.617882 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:43.619739 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:43.619745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:43.619919 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:43.619951 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 18:46:43.620101 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 18:46:43.620241 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11893,"thread_start_us":11591,"threads_started":1}
I20260620 18:46:43.621577 25057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d08ae625f9514546a853913ae1564b6c" candidate_uuid: "76c804b22c6b4a80aa71ee99553c918b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "567bacc61bc0415586646dba2169b956" is_pre_election: true
I20260620 18:46:43.621613 25256 raft_consensus.cc:504] T d08ae625f9514546a853913ae1564b6c P 76c804b22c6b4a80aa71ee99553c918b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617a7f264d364698a3b05b588c7ed2b8" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 45531 } } peers { permanent_uuid: "567bacc61bc0415586646dba2169b956" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39991 } } peers { permanent_uuid: "76c804b22c6b4a80aa71ee99553c918b" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 35701 } }
I20260620 18:46:43.622061 25057 raft_consensus.cc:2379] T d08ae625f9514546a853913ae1564b6c P 567bacc61bc0415586646dba2169b956 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 76c804b22c6b4a80aa71ee99553c918b in current term 1: Already voted for candidate 567bacc61bc0415586646dba2169b956 in this term.
I20260620 18:46:43.622100 25187 raft_consensus.cc:2379] T d08ae625f9514546a853913ae1564b6c P 76c804b22c6b4a80aa71ee99553c918b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 567bacc61bc0415586646dba2169b956 in current term 1: Already voted for candidate 76c804b22c6b4a80aa71ee99553c918b in this term.
I20260620 18:46:43.622337 25256 leader_election.cc:288] T d08ae625f9514546a853913ae1564b6c P 76c804b22c6b4a80aa71ee99553c918b [CANDIDATE]: Term 1 election: Requested vote from peers 617a7f264d364698a3b05b588c7ed2b8 (127.5.225.129:45531), 567bacc61bc0415586646dba2169b956 (127.5.225.130:39991)
I20260620 18:46:43.622915 24926 raft_consensus.cc:2454] T d08ae625f9514546a853913ae1564b6c P 617a7f264d364698a3b05b588c7ed2b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 567bacc61bc0415586646dba2169b956 in term 1.
I20260620 18:46:43.623138 25057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d08ae625f9514546a853913ae1564b6c" candidate_uuid: "76c804b22c6b4a80aa71ee99553c918b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "567bacc61bc0415586646dba2169b956"
I20260620 18:46:43.623152 24925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d08ae625f9514546a853913ae1564b6c" candidate_uuid: "76c804b22c6b4a80aa71ee99553c918b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "617a7f264d364698a3b05b588c7ed2b8"
I20260620 18:46:43.623314 24993 leader_election.cc:302] T d08ae625f9514546a853913ae1564b6c P 567bacc61bc0415586646dba2169b956 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 567bacc61bc0415586646dba2169b956, 617a7f264d364698a3b05b588c7ed2b8; no voters:
I20260620 18:46:43.623400 25057 raft_consensus.cc:2379] T d08ae625f9514546a853913ae1564b6c P 567bacc61bc0415586646dba2169b956 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 76c804b22c6b4a80aa71ee99553c918b in current term 1: Already voted for candidate 567bacc61bc0415586646dba2169b956 in this term.
I20260620 18:46:43.623456 24925 raft_consensus.cc:2379] T d08ae625f9514546a853913ae1564b6c P 617a7f264d364698a3b05b588c7ed2b8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 76c804b22c6b4a80aa71ee99553c918b in current term 1: Already voted for candidate 567bacc61bc0415586646dba2169b956 in this term.
I20260620 18:46:43.623972 25257 raft_consensus.cc:2781] T d08ae625f9514546a853913ae1564b6c P 567bacc61bc0415586646dba2169b956 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:46:43.624027 25123 leader_election.cc:302] T d08ae625f9514546a853913ae1564b6c P 76c804b22c6b4a80aa71ee99553c918b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76c804b22c6b4a80aa71ee99553c918b; no voters: 567bacc61bc0415586646dba2169b956, 617a7f264d364698a3b05b588c7ed2b8
I20260620 18:46:43.624522 25257 raft_consensus.cc:686] T d08ae625f9514546a853913ae1564b6c P 567bacc61bc0415586646dba2169b956 [term 1 LEADER]: Becoming Leader. State: Replica: 567bacc61bc0415586646dba2169b956, State: Running, Role: LEADER
I20260620 18:46:43.624568 25256 raft_consensus.cc:2726] T d08ae625f9514546a853913ae1564b6c P 76c804b22c6b4a80aa71ee99553c918b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260620 18:46:43.625101 25257 consensus_queue.cc:237] T d08ae625f9514546a853913ae1564b6c P 567bacc61bc0415586646dba2169b956 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617a7f264d364698a3b05b588c7ed2b8" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 45531 } } peers { permanent_uuid: "567bacc61bc0415586646dba2169b956" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39991 } } peers { permanent_uuid: "76c804b22c6b4a80aa71ee99553c918b" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 35701 } }
I20260620 18:46:43.629403 24785 catalog_manager.cc:5495] T d08ae625f9514546a853913ae1564b6c P 567bacc61bc0415586646dba2169b956 reported cstate change: term changed from 0 to 1, leader changed from <none> to 567bacc61bc0415586646dba2169b956 (127.5.225.130). New cstate: current_term: 1 leader_uuid: "567bacc61bc0415586646dba2169b956" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617a7f264d364698a3b05b588c7ed2b8" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 45531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "567bacc61bc0415586646dba2169b956" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 39991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76c804b22c6b4a80aa71ee99553c918b" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 35701 } health_report { overall_health: UNKNOWN } } }
I20260620 18:46:43.700238 25247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:43.694776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57212 (local address 127.5.225.130:39991)
0620 18:46:43.694942 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:43.694947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:43.694964 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:46:43.695140 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:43.695144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:43.695229 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:43.695355 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:46:43.695739 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:43.696472 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:43.697943 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:43.698497 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:43.698596 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:43.699139 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:46:43.699700 (+ 561us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:46:43.699829 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:43.699977 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 18:46:43.700014 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:46:43.708673 24925 raft_consensus.cc:1261] T d08ae625f9514546a853913ae1564b6c P 617a7f264d364698a3b05b588c7ed2b8 [term 1 FOLLOWER]: Refusing update from remote peer 567bacc61bc0415586646dba2169b956: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:46:43.708673 25187 raft_consensus.cc:1261] T d08ae625f9514546a853913ae1564b6c P 76c804b22c6b4a80aa71ee99553c918b [term 1 FOLLOWER]: Refusing update from remote peer 567bacc61bc0415586646dba2169b956: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:46:43.709568 25263 consensus_queue.cc:1035] T d08ae625f9514546a853913ae1564b6c P 567bacc61bc0415586646dba2169b956 [LEADER]: Connected to new peer: Peer: permanent_uuid: "617a7f264d364698a3b05b588c7ed2b8" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 45531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:46:43.709926 25257 consensus_queue.cc:1035] T d08ae625f9514546a853913ae1564b6c P 567bacc61bc0415586646dba2169b956 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c804b22c6b4a80aa71ee99553c918b" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 35701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:46:43.737175 24785 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:46:43.737447 24785 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:46:43.740926 24785 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4cdc78914544465ea6fbadf99e5ccdbb: Sending DeleteTablet for 3 replicas of tablet d08ae625f9514546a853913ae1564b6c
I20260620 18:46:43.741995 24906 tablet_service.cc:1514] Processing DeleteTablet for tablet d08ae625f9514546a853913ae1564b6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:46:43 UTC) from {username='slave'} at 127.0.0.1:50676
I20260620 18:46:43.742477 25037 tablet_service.cc:1514] Processing DeleteTablet for tablet d08ae625f9514546a853913ae1564b6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:46:43 UTC) from {username='slave'} at 127.0.0.1:57202
I20260620 18:46:43.743295 25167 tablet_service.cc:1514] Processing DeleteTablet for tablet d08ae625f9514546a853913ae1564b6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:46:43 UTC) from {username='slave'} at 127.0.0.1:42192
I20260620 18:46:43.743417 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 24844
I20260620 18:46:43.764648 25245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:43.744747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50688 (local address 127.5.225.129:45531)
0620 18:46:43.744938 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:43.744945 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:43.755791 (+ 10846us) server_negotiation.cc:407] Connection header received
0620 18:46:43.756261 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:43.756268 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:43.756379 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:43.756466 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:43.757207 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:43.758434 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:43.760442 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:43.761071 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:43.761169 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:43.762800 (+ 1631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:43.762814 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:43.762833 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:46:43.763708 (+ 875us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:43.763773 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:46:43.763797 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:43.763884 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:43.764130 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 18:46:43.764245 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:46:43.766198 24966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:46:43.768630 25278 tablet_replica.cc:330] T d08ae625f9514546a853913ae1564b6c P 76c804b22c6b4a80aa71ee99553c918b: stopping tablet replica
I20260620 18:46:43.770049 25278 raft_consensus.cc:2229] T d08ae625f9514546a853913ae1564b6c P 76c804b22c6b4a80aa71ee99553c918b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:46:43.770696 25277 tablet_replica.cc:330] T d08ae625f9514546a853913ae1564b6c P 567bacc61bc0415586646dba2169b956: stopping tablet replica
I20260620 18:46:43.771338 25277 raft_consensus.cc:2229] T d08ae625f9514546a853913ae1564b6c P 567bacc61bc0415586646dba2169b956 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:46:43.772157 25277 raft_consensus.cc:2258] T d08ae625f9514546a853913ae1564b6c P 567bacc61bc0415586646dba2169b956 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:46:43.772930 25278 raft_consensus.cc:2258] T d08ae625f9514546a853913ae1564b6c P 76c804b22c6b4a80aa71ee99553c918b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:46:43.775828 25278 ts_tablet_manager.cc:1905] T d08ae625f9514546a853913ae1564b6c P 76c804b22c6b4a80aa71ee99553c918b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:46:43.780449 25277 ts_tablet_manager.cc:1905] T d08ae625f9514546a853913ae1564b6c P 567bacc61bc0415586646dba2169b956: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:46:43.787772 25278 ts_tablet_manager.cc:1918] T d08ae625f9514546a853913ae1564b6c P 76c804b22c6b4a80aa71ee99553c918b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:46:43.788103 25278 log.cc:1196] T d08ae625f9514546a853913ae1564b6c P 76c804b22c6b4a80aa71ee99553c918b: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-2/wal/wals/d08ae625f9514546a853913ae1564b6c
I20260620 18:46:43.788359 25277 ts_tablet_manager.cc:1918] T d08ae625f9514546a853913ae1564b6c P 567bacc61bc0415586646dba2169b956: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:46:43.788559 25277 log.cc:1196] T d08ae625f9514546a853913ae1564b6c P 567bacc61bc0415586646dba2169b956: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980894474392-6022-0/minicluster-data/ts-1/wal/wals/d08ae625f9514546a853913ae1564b6c
I20260620 18:46:43.788887 25278 ts_tablet_manager.cc:1939] T d08ae625f9514546a853913ae1564b6c P 76c804b22c6b4a80aa71ee99553c918b: Deleting consensus metadata
I20260620 18:46:43.789155 25277 ts_tablet_manager.cc:1939] T d08ae625f9514546a853913ae1564b6c P 567bacc61bc0415586646dba2169b956: Deleting consensus metadata
I20260620 18:46:43.791016 24772 catalog_manager.cc:4841] TS 76c804b22c6b4a80aa71ee99553c918b (127.5.225.131:35701): tablet d08ae625f9514546a853913ae1564b6c (table test-table [id=d484d00cd19147699e7198ec80a6e98a]) successfully deleted
I20260620 18:46:43.791361 24772 catalog_manager.cc:4841] TS 567bacc61bc0415586646dba2169b956 (127.5.225.130:39991): tablet d08ae625f9514546a853913ae1564b6c (table test-table [id=d484d00cd19147699e7198ec80a6e98a]) successfully deleted
I20260620 18:46:43.887969 25276 tablet_replica.cc:330] T d08ae625f9514546a853913ae1564b6c P 617a7f264d364698a3b05b588c7ed2b8: stopping tablet replica
I20260620 18:46:43.888034 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 24844
I20260620 18:46:43.905786 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 24975
W20260620 18:46:43.905709 24772 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.225.129:45531 (error 108)
I20260620 18:46:43.907819 25282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:43.907004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:45531 (local address 127.0.0.1:50690)
0620 18:46:43.907500 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:43.907656 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.129:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":261,"threads_started":1}
W20260620 18:46:43.908125 24772 catalog_manager.cc:4584] TS 617a7f264d364698a3b05b588c7ed2b8 (127.5.225.129:45531): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d08ae625f9514546a853913ae1564b6c: Network error: Client connection negotiation failed: client connection to 127.5.225.129:45531: connect: Connection refused (error 111)
I20260620 18:46:43.913851 25247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:43.906712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57220 (local address 127.5.225.130:39991)
0620 18:46:43.906893 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:43.906901 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:43.906965 (+ 64us) server_negotiation.cc:407] Connection header received
0620 18:46:43.907194 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:43.907201 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:43.907330 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:43.907439 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:43.908005 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:43.908886 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:43.910779 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:43.911183 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:43.911284 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:43.912470 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:43.912479 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:43.912492 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:46:43.913085 (+ 593us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:43.913130 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:46:43.913148 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:43.913230 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:43.913394 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 18:46:43.913479 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 18:46:43.914564 25097 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:46:43.961691 25282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:43.961156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:45531 (local address 127.0.0.1:50702)
0620 18:46:43.961389 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:43.961509 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.129:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 18:46:43.962824 25282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:43.962398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:45531 (local address 127.0.0.1:50710)
0620 18:46:43.962581 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:43.962667 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.129:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 18:46:44.023677 25282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:44.023124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:45531 (local address 127.0.0.1:50720)
0620 18:46:44.023372 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:44.023485 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.129:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260620 18:46:44.024888 25282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:44.024442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:45531 (local address 127.0.0.1:50730)
0620 18:46:44.024618 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:44.024704 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.129:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 18:46:44.027805 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 24975
I20260620 18:46:44.047066 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 25105
I20260620 18:46:44.053830 25249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:44.047927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42198 (local address 127.5.225.131:35701)
0620 18:46:44.048121 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:44.048125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:44.048145 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:46:44.048321 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:44.048325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:44.048420 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:44.048501 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:44.048864 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:44.049565 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:44.050947 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:44.051341 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:44.051438 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:44.052397 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:44.052406 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:44.052423 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:46:44.053007 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:44.053052 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:46:44.053071 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:44.053159 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:44.053406 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 18:46:44.053488 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 18:46:44.054538 25227 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:46:44.105571 25282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:44.104964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:45531 (local address 127.0.0.1:50732)
0620 18:46:44.105235 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:44.105369 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.129:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260620 18:46:44.106913 25282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:44.106374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:45531 (local address 127.0.0.1:50740)
0620 18:46:44.106656 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:44.106745 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.129:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260620 18:46:44.144760 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 25105
I20260620 18:46:44.161624 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 24754
I20260620 18:46:44.162740 24815 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:46:44.271982 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 24754
2026-06-20T18:46:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2966 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:46:44 dist-test-slave-k6lg krb5kdc[25288](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:46:44 dist-test-slave-k6lg krb5kdc[25288](info): set up 2 sockets
Jun 20 18:46:44 dist-test-slave-k6lg krb5kdc[25288](info): commencing operation
krb5kdc: starting...
W20260620 18:46:45.394546 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:46:45 dist-test-slave-k6lg krb5kdc[25288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:46:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:46:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
I20260620 18:46:45.645313 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:36747
--webserver_interface=127.5.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:40743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:36747
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:46:45.970070 25301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:46:45.970414 25301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:46:45.970518 25301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:46:45.977538 25301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:46:45.977641 25301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:46:45.977687 25301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:46:45.977744 25301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:46:45.977787 25301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:46:45.987527 25301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:40743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:36747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:36747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:46:45.988794 25301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:46:45.990067 25301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:46:46.002465 25306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:46.002403 25309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:46.002487 25307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:46:46.003732 25301 server_base.cc:1034] running on GCE node
I20260620 18:46:46.004601 25301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:46:46.006098 25301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:46:46.007375 25301 hybrid_clock.cc:648] HybridClock initialized: now 1781981206007321 us; error 40 us; skew 500 ppm
I20260620 18:46:46.007728 25301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:46:46 dist-test-slave-k6lg krb5kdc[25288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981206, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:46:46.010756 25301 init.cc:377] Logged in from keytab as kudu/127.5.225.190@KRBTEST.COM (short username kudu)
I20260620 18:46:46.012282 25301 webserver.cc:460] Webserver started at http://127.5.225.190:41991/ using document root <none> and password file <none>
I20260620 18:46:46.012949 25301 fs_manager.cc:359] Metadata directory not provided
I20260620 18:46:46.013103 25301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:46:46.013402 25301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:46:46.016353 25301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "273b3f828ea84638a353c6aa059704fa"
format_stamp: "Formatted at 2026-06-20 18:46:46 on dist-test-slave-k6lg"
server_key: "689228c3ccc1200e203170fa5cb325ce"
server_key_iv: "1dcbdde4f0a79b30481761ebbfdbaf88"
server_key_version: "encryptionkey@0"
I20260620 18:46:46.017132 25301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "273b3f828ea84638a353c6aa059704fa"
format_stamp: "Formatted at 2026-06-20 18:46:46 on dist-test-slave-k6lg"
server_key: "689228c3ccc1200e203170fa5cb325ce"
server_key_iv: "1dcbdde4f0a79b30481761ebbfdbaf88"
server_key_version: "encryptionkey@0"
I20260620 18:46:46.022100 25301 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260620 18:46:46.025445 25316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:46.026811 25301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:46:46.027009 25301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "273b3f828ea84638a353c6aa059704fa"
format_stamp: "Formatted at 2026-06-20 18:46:46 on dist-test-slave-k6lg"
server_key: "689228c3ccc1200e203170fa5cb325ce"
server_key_iv: "1dcbdde4f0a79b30481761ebbfdbaf88"
server_key_version: "encryptionkey@0"
I20260620 18:46:46.027210 25301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:46:46.042413 25301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:46:46.046322 25301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:46:46.046608 25301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:46:46.065557 25301 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:36747
I20260620 18:46:46.065563 25367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:36747 every 8 connection(s)
I20260620 18:46:46.067018 25301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:46:46.068169 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 25301
I20260620 18:46:46.068365 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:46:46.068897 6022 external_mini_cluster.cc:1428] Setting key 42b802e9e6eb0a240a1b5ad076990fe4
I20260620 18:46:46.071770 25368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 18:46:46 dist-test-slave-k6lg krb5kdc[25288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:46:46.085032 25368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 273b3f828ea84638a353c6aa059704fa: Bootstrap starting.
I20260620 18:46:46.088647 25368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 273b3f828ea84638a353c6aa059704fa: Neither blocks nor log segments found. Creating new log.
I20260620 18:46:46.089671 25368 log.cc:824] T 00000000000000000000000000000000 P 273b3f828ea84638a353c6aa059704fa: Log is configured to *not* fsync() on all Append() calls
I20260620 18:46:46.092998 25368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 273b3f828ea84638a353c6aa059704fa: No bootstrap required, opened a new log
I20260620 18:46:46.093653 25371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:46.070803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33348 (local address 127.5.225.190:36747)
0620 18:46:46.071615 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:46.071627 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:46.071669 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:46:46.073331 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:46.073353 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:46.073727 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:46.074048 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:46.075886 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:46.077310 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:46.079175 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:46.079848 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:46.079979 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:46.085084 (+ 5105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:46.085093 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:46.085109 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:46:46.085155 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:46.089034 (+ 3879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:46.089905 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:46.089912 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:46.089918 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:46.090085 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:46.090646 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:46.090652 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:46.090656 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:46.091366 (+ 710us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:46.091592 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:46.092136 (+ 544us) server_negotiation.cc:299] Negotiation successful
0620 18:46:46.092547 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":357,"threads_started":1}
I20260620 18:46:46.100575 25368 raft_consensus.cc:348] T 00000000000000000000000000000000 P 273b3f828ea84638a353c6aa059704fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273b3f828ea84638a353c6aa059704fa" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36747 } }
I20260620 18:46:46.100960 25368 raft_consensus.cc:374] T 00000000000000000000000000000000 P 273b3f828ea84638a353c6aa059704fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:46:46.101068 25368 raft_consensus.cc:729] T 00000000000000000000000000000000 P 273b3f828ea84638a353c6aa059704fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 273b3f828ea84638a353c6aa059704fa, State: Initialized, Role: FOLLOWER
I20260620 18:46:46.101586 25368 consensus_queue.cc:260] T 00000000000000000000000000000000 P 273b3f828ea84638a353c6aa059704fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273b3f828ea84638a353c6aa059704fa" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36747 } }
I20260620 18:46:46.101838 25368 raft_consensus.cc:388] T 00000000000000000000000000000000 P 273b3f828ea84638a353c6aa059704fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:46:46.101986 25368 raft_consensus.cc:482] T 00000000000000000000000000000000 P 273b3f828ea84638a353c6aa059704fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:46:46.102216 25368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 273b3f828ea84638a353c6aa059704fa [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:46:46.104350 25368 raft_consensus.cc:504] T 00000000000000000000000000000000 P 273b3f828ea84638a353c6aa059704fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273b3f828ea84638a353c6aa059704fa" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36747 } }
I20260620 18:46:46.104853 25368 leader_election.cc:302] T 00000000000000000000000000000000 P 273b3f828ea84638a353c6aa059704fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 273b3f828ea84638a353c6aa059704fa; no voters:
I20260620 18:46:46.105428 25368 leader_election.cc:288] T 00000000000000000000000000000000 P 273b3f828ea84638a353c6aa059704fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:46:46.105561 25373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 273b3f828ea84638a353c6aa059704fa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:46:46.106324 25373 raft_consensus.cc:686] T 00000000000000000000000000000000 P 273b3f828ea84638a353c6aa059704fa [term 1 LEADER]: Becoming Leader. State: Replica: 273b3f828ea84638a353c6aa059704fa, State: Running, Role: LEADER
I20260620 18:46:46.106864 25373 consensus_queue.cc:237] T 00000000000000000000000000000000 P 273b3f828ea84638a353c6aa059704fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273b3f828ea84638a353c6aa059704fa" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36747 } }
I20260620 18:46:46.107556 25368 sys_catalog.cc:564] T 00000000000000000000000000000000 P 273b3f828ea84638a353c6aa059704fa [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:46:46.109848 25374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 273b3f828ea84638a353c6aa059704fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "273b3f828ea84638a353c6aa059704fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273b3f828ea84638a353c6aa059704fa" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36747 } } }
I20260620 18:46:46.110302 25374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 273b3f828ea84638a353c6aa059704fa [sys.catalog]: This master's current role is: LEADER
I20260620 18:46:46.110212 25375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 273b3f828ea84638a353c6aa059704fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 273b3f828ea84638a353c6aa059704fa. Latest consensus state: current_term: 1 leader_uuid: "273b3f828ea84638a353c6aa059704fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273b3f828ea84638a353c6aa059704fa" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36747 } } }
I20260620 18:46:46.110493 25375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 273b3f828ea84638a353c6aa059704fa [sys.catalog]: This master's current role is: LEADER
I20260620 18:46:46.112565 25381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:46:46.117652 25381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:46:46.126926 25381 catalog_manager.cc:1347] Generated new cluster ID: 6a10cdb186c4420d94dec0759bb0cca4
I20260620 18:46:46.127070 25381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:46:46.142769 25381 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:46:46.144057 25381 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:46:46.155252 25381 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 273b3f828ea84638a353c6aa059704fa: Generated new TSK 0
I20260620 18:46:46.156337 25381 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab.
Entry for principal kudu/127.5.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab.
I20260620 18:46:46.221268 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.129:0
--local_ip_for_outbound_sockets=127.5.225.129
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:36747
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:40743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:46:46.559486 25394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:46:46.559819 25394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:46:46.559991 25394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:46:46.567224 25394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:46:46.567391 25394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:46:46.567569 25394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.129
I20260620 18:46:46.577265 25394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:40743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.129.keytab
--principal=kudu/127.5.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:36747
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.129
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:46:46.578622 25394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:46:46.579964 25394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:46:46.593169 25399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:46.593209 25402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:46.593232 25400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:46:46.594398 25394 server_base.cc:1034] running on GCE node
I20260620 18:46:46.595093 25394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:46:46.596004 25394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:46:46.597203 25394 hybrid_clock.cc:648] HybridClock initialized: now 1781981206597143 us; error 49 us; skew 500 ppm
I20260620 18:46:46.597510 25394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:46:46 dist-test-slave-k6lg krb5kdc[25288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981206, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:46:46.600615 25394 init.cc:377] Logged in from keytab as kudu/127.5.225.129@KRBTEST.COM (short username kudu)
I20260620 18:46:46.601958 25394 webserver.cc:460] Webserver started at http://127.5.225.129:40677/ using document root <none> and password file <none>
I20260620 18:46:46.602671 25394 fs_manager.cc:359] Metadata directory not provided
I20260620 18:46:46.602805 25394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:46:46.603127 25394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:46:46.605963 25394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-0/data/instance:
uuid: "a503992b294745618bb796a526245cce"
format_stamp: "Formatted at 2026-06-20 18:46:46 on dist-test-slave-k6lg"
server_key: "8af44f2df2635fe2fa0d8819fc6e77cb"
server_key_iv: "577df14cc41e6090ee9e39d5e2ba8ff6"
server_key_version: "encryptionkey@0"
I20260620 18:46:46.606812 25394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance:
uuid: "a503992b294745618bb796a526245cce"
format_stamp: "Formatted at 2026-06-20 18:46:46 on dist-test-slave-k6lg"
server_key: "8af44f2df2635fe2fa0d8819fc6e77cb"
server_key_iv: "577df14cc41e6090ee9e39d5e2ba8ff6"
server_key_version: "encryptionkey@0"
I20260620 18:46:46.612435 25394 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 18:46:46.615999 25409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:46.617296 25394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:46:46.617468 25394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-0/wal
uuid: "a503992b294745618bb796a526245cce"
format_stamp: "Formatted at 2026-06-20 18:46:46 on dist-test-slave-k6lg"
server_key: "8af44f2df2635fe2fa0d8819fc6e77cb"
server_key_iv: "577df14cc41e6090ee9e39d5e2ba8ff6"
server_key_version: "encryptionkey@0"
I20260620 18:46:46.617671 25394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:46:46.635144 25394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:46:46.639444 25394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:46:46.639742 25394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:46:46.640668 25394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:46:46.641973 25394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:46:46.642045 25394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:46.642246 25394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:46:46.642300 25394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:46.675230 25394 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.129:40337
I20260620 18:46:46.675318 25521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.129:40337 every 8 connection(s)
I20260620 18:46:46.676545 25394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
I20260620 18:46:46.680166 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 25394
I20260620 18:46:46.680352 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance
I20260620 18:46:46.680869 6022 external_mini_cluster.cc:1428] Setting key a0de6507d84975c8d027a233d6445de1
Jun 20 18:46:46 dist-test-slave-k6lg krb5kdc[25288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981206, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.129@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:46:46.697613 25524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:46.679155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:47353 (local address 127.5.225.190:36747)
0620 18:46:46.679578 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:46.679582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:46.680738 (+ 1156us) server_negotiation.cc:407] Connection header received
0620 18:46:46.682854 (+ 2116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:46.682860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:46.682947 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:46.683065 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:46.684522 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:46.685247 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:46.686880 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:46.687292 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:46.687433 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:46.690899 (+ 3466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:46.690909 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:46.690917 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:46:46.690963 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:46.694445 (+ 3482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:46.695126 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:46.695140 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:46.695147 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:46.695303 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:46.695704 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:46.695712 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:46.695717 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:46.696080 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:46.696238 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:46.696893 (+ 655us) server_negotiation.cc:299] Negotiation successful
0620 18:46:46.697151 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":214,"threads_started":1}
I20260620 18:46:46.698240 25525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:46.679716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36747 (local address 127.5.225.129:47353)
0620 18:46:46.680568 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:46.680607 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:46.682594 (+ 1987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:46.683307 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:46.683315 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:46.683814 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:46:46.684351 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:46.684369 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:46.685407 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:46.685412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:46.686738 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:46.686749 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:46.687523 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:46.687532 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:46.687772 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:46.688460 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:46:46.688485 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:46:46.690688 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:46:46.694581 (+ 3893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:46:46.694588 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:46:46.694599 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:46:46.694994 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:46:46.695412 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:46:46.695415 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:46:46.695417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:46:46.695584 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:46:46.696214 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:46:46.696220 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:46:46.696506 (+ 286us) client_negotiation.cc:769] Sending connection context
0620 18:46:46.696727 (+ 221us) client_negotiation.cc:240] Negotiation successful
0620 18:46:46.697088 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":320,"threads_started":1}
I20260620 18:46:46.699801 25522 heartbeater.cc:347] Connected to a master server at 127.5.225.190:36747
I20260620 18:46:46.700116 25522 heartbeater.cc:464] Registering TS with master...
I20260620 18:46:46.700973 25522 heartbeater.cc:511] Master 127.5.225.190:36747 requested a full tablet report, sending...
I20260620 18:46:46.703233 25333 ts_manager.cc:194] Registered new tserver with Master: a503992b294745618bb796a526245cce (127.5.225.129:40337)
I20260620 18:46:46.704653 25333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.129@KRBTEST.COM'} at 127.5.225.129:47353
WARNING: no policy specified for kudu/127.5.225.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab.
Entry for principal kudu/127.5.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab.
I20260620 18:46:46.737267 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.130:0
--local_ip_for_outbound_sockets=127.5.225.130
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:36747
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:40743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:46:47.063490 25528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:46:47.063828 25528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:46:47.064003 25528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:46:47.075079 25528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:46:47.075227 25528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:46:47.075455 25528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.130
I20260620 18:46:47.087692 25528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:40743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.130.keytab
--principal=kudu/127.5.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:36747
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.130
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:46:47.088898 25528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:46:47.090240 25528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:46:47.104171 25534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:47.104133 25533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:47.104211 25536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:46:47.104908 25528 server_base.cc:1034] running on GCE node
I20260620 18:46:47.105604 25528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:46:47.106740 25528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:46:47.108007 25528 hybrid_clock.cc:648] HybridClock initialized: now 1781981207107953 us; error 38 us; skew 500 ppm
I20260620 18:46:47.108397 25528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:46:47 dist-test-slave-k6lg krb5kdc[25288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981207, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:46:47.115103 25528 init.cc:377] Logged in from keytab as kudu/127.5.225.130@KRBTEST.COM (short username kudu)
I20260620 18:46:47.116829 25528 webserver.cc:460] Webserver started at http://127.5.225.130:39865/ using document root <none> and password file <none>
I20260620 18:46:47.117560 25528 fs_manager.cc:359] Metadata directory not provided
I20260620 18:46:47.117694 25528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:46:47.118043 25528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:46:47.121282 25528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-1/data/instance:
uuid: "316f0723e41445ddb5d3cd8e1e456eb0"
format_stamp: "Formatted at 2026-06-20 18:46:47 on dist-test-slave-k6lg"
server_key: "d69c247bf56f13d9cc436afb9298ef90"
server_key_iv: "c94093374b2aed3f2149c624e3e737aa"
server_key_version: "encryptionkey@0"
I20260620 18:46:47.122115 25528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance:
uuid: "316f0723e41445ddb5d3cd8e1e456eb0"
format_stamp: "Formatted at 2026-06-20 18:46:47 on dist-test-slave-k6lg"
server_key: "d69c247bf56f13d9cc436afb9298ef90"
server_key_iv: "c94093374b2aed3f2149c624e3e737aa"
server_key_version: "encryptionkey@0"
I20260620 18:46:47.127455 25528 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:46:47.131461 25543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:47.132887 25528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:46:47.133076 25528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-1/wal
uuid: "316f0723e41445ddb5d3cd8e1e456eb0"
format_stamp: "Formatted at 2026-06-20 18:46:47 on dist-test-slave-k6lg"
server_key: "d69c247bf56f13d9cc436afb9298ef90"
server_key_iv: "c94093374b2aed3f2149c624e3e737aa"
server_key_version: "encryptionkey@0"
I20260620 18:46:47.133288 25528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:46:47.147755 25528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:46:47.152489 25528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:46:47.152796 25528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:46:47.153800 25528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:46:47.155233 25528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:46:47.155364 25528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:47.155500 25528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:46:47.155562 25528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:47.187165 25528 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.130:33413
I20260620 18:46:47.187206 25655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.130:33413 every 8 connection(s)
I20260620 18:46:47.188500 25528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
I20260620 18:46:47.191061 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 25528
I20260620 18:46:47.191248 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance
I20260620 18:46:47.191715 6022 external_mini_cluster.cc:1428] Setting key fcb60e51df4539f3e66940d1b8b2c5ba
Jun 20 18:46:47 dist-test-slave-k6lg krb5kdc[25288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981207, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.130@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:46:47.211673 25524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:47.192222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:44649 (local address 127.5.225.190:36747)
0620 18:46:47.192414 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:47.192420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:47.193105 (+ 685us) server_negotiation.cc:407] Connection header received
0620 18:46:47.195072 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:47.195080 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:47.195175 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:47.195291 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:47.196801 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:47.197906 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:47.199814 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:47.200573 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:47.200721 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:47.205227 (+ 4506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:47.205233 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:47.205236 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:46:47.205270 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:47.208098 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:47.209162 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:47.209165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:47.209168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:47.209290 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:47.209800 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:47.209803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:47.209805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:47.210119 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:47.210318 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:47.211237 (+ 919us) server_negotiation.cc:299] Negotiation successful
0620 18:46:47.211424 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:46:47.213295 25658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:47.191779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36747 (local address 127.5.225.130:44649)
0620 18:46:47.192649 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:47.192694 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:47.194796 (+ 2102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:47.195491 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:47.195498 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:47.196043 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:46:47.196622 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:47.196639 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:47.198135 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:47.198141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:47.199666 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:47.199678 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:47.200812 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:47.200818 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:47.201037 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:47.201791 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:46:47.201814 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:46:47.204954 (+ 3140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:46:47.208298 (+ 3344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:46:47.208330 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:46:47.208352 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:46:47.209036 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:46:47.209400 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:46:47.209407 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:46:47.209411 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:46:47.209681 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:46:47.210318 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:46:47.210329 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:46:47.210808 (+ 479us) client_negotiation.cc:769] Sending connection context
0620 18:46:47.211121 (+ 313us) client_negotiation.cc:240] Negotiation successful
0620 18:46:47.211715 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":415,"threads_started":1}
I20260620 18:46:47.214937 25656 heartbeater.cc:347] Connected to a master server at 127.5.225.190:36747
I20260620 18:46:47.215317 25656 heartbeater.cc:464] Registering TS with master...
I20260620 18:46:47.216331 25656 heartbeater.cc:511] Master 127.5.225.190:36747 requested a full tablet report, sending...
I20260620 18:46:47.217993 25333 ts_manager.cc:194] Registered new tserver with Master: 316f0723e41445ddb5d3cd8e1e456eb0 (127.5.225.130:33413)
WARNING: no policy specified for kudu/127.5.225.131@KRBTEST.COM; defaulting to no policy
I20260620 18:46:47.218921 25333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.130@KRBTEST.COM'} at 127.5.225.130:44649
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab.
Entry for principal kudu/127.5.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab.
I20260620 18:46:47.249311 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.131:0
--local_ip_for_outbound_sockets=127.5.225.131
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:36747
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:40743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:46:47.573987 25661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:46:47.574277 25661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:46:47.574374 25661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:46:47.581542 25661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:46:47.581641 25661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:46:47.581753 25661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.131
I20260620 18:46:47.591663 25661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:40743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.131.keytab
--principal=kudu/127.5.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:36747
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.131
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:46:47.592882 25661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:46:47.594152 25661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:46:47.607394 25666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:47.607421 25667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:47.607338 25669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:46:47.608462 25661 server_base.cc:1034] running on GCE node
I20260620 18:46:47.609151 25661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:46:47.610111 25661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:46:47.611404 25661 hybrid_clock.cc:648] HybridClock initialized: now 1781981207611352 us; error 35 us; skew 500 ppm
I20260620 18:46:47.611763 25661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:46:47 dist-test-slave-k6lg krb5kdc[25288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981207, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:46:47.615375 25661 init.cc:377] Logged in from keytab as kudu/127.5.225.131@KRBTEST.COM (short username kudu)
I20260620 18:46:47.616839 25661 webserver.cc:460] Webserver started at http://127.5.225.131:46337/ using document root <none> and password file <none>
I20260620 18:46:47.617523 25661 fs_manager.cc:359] Metadata directory not provided
I20260620 18:46:47.617671 25661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:46:47.617982 25661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:46:47.621002 25661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-2/data/instance:
uuid: "a3496e4323f941ae9e15dcb6782bbb09"
format_stamp: "Formatted at 2026-06-20 18:46:47 on dist-test-slave-k6lg"
server_key: "f7ea602ada184e1f71cd777d0dad7610"
server_key_iv: "74612c60e47fb829c6ee82d2a4723d3b"
server_key_version: "encryptionkey@0"
I20260620 18:46:47.621892 25661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance:
uuid: "a3496e4323f941ae9e15dcb6782bbb09"
format_stamp: "Formatted at 2026-06-20 18:46:47 on dist-test-slave-k6lg"
server_key: "f7ea602ada184e1f71cd777d0dad7610"
server_key_iv: "74612c60e47fb829c6ee82d2a4723d3b"
server_key_version: "encryptionkey@0"
I20260620 18:46:47.626933 25661 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.003s
I20260620 18:46:47.630295 25676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:47.631667 25661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:46:47.631844 25661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-2/wal
uuid: "a3496e4323f941ae9e15dcb6782bbb09"
format_stamp: "Formatted at 2026-06-20 18:46:47 on dist-test-slave-k6lg"
server_key: "f7ea602ada184e1f71cd777d0dad7610"
server_key_iv: "74612c60e47fb829c6ee82d2a4723d3b"
server_key_version: "encryptionkey@0"
I20260620 18:46:47.632050 25661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:46:47.650544 25661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:46:47.654915 25661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:46:47.655223 25661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:46:47.656168 25661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:46:47.657510 25661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:46:47.657612 25661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:47.657768 25661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:46:47.657851 25661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:47.691668 25661 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.131:43335
I20260620 18:46:47.691741 25788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.131:43335 every 8 connection(s)
I20260620 18:46:47.692951 25661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
I20260620 18:46:47.703183 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 25661
I20260620 18:46:47.703385 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance
I20260620 18:46:47.703832 6022 external_mini_cluster.cc:1428] Setting key ddc04a00f03264355be75d5727875c3a
Jun 20 18:46:47 dist-test-slave-k6lg krb5kdc[25288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981207, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.131@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:46:47.707954 25522 heartbeater.cc:503] Master 127.5.225.190:36747 was elected leader, sending a full tablet report...
I20260620 18:46:47.713344 25524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:47.695401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:58993 (local address 127.5.225.190:36747)
0620 18:46:47.695592 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:47.695596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:47.696611 (+ 1015us) server_negotiation.cc:407] Connection header received
0620 18:46:47.698474 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:47.698480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:47.698588 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:47.698705 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:47.700142 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:47.700879 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:47.702461 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:47.702859 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:47.703004 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:47.707090 (+ 4086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:47.707094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:47.707098 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:46:47.707129 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:47.710420 (+ 3291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:47.711116 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:47.711120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:47.711123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:47.711250 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:47.711748 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:47.711753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:47.711755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:47.712043 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:47.712174 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:47.712879 (+ 705us) server_negotiation.cc:299] Negotiation successful
0620 18:46:47.713094 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:46:47.714417 25791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:47.695780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:36747 (local address 127.5.225.131:58993)
0620 18:46:47.696443 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:47.696488 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:47.698128 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:47.698940 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:47.698947 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:47.699429 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:46:47.699968 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:47.699985 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:47.701033 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:47.701039 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:47.702324 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:47.702335 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:47.703090 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:47.703099 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:47.703373 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:47.704139 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:46:47.704166 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:46:47.706852 (+ 2686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:46:47.710540 (+ 3688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:46:47.710547 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:46:47.710560 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:46:47.710994 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:46:47.711375 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:46:47.711379 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:46:47.711381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:46:47.711606 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:46:47.712186 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:46:47.712192 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:46:47.712510 (+ 318us) client_negotiation.cc:769] Sending connection context
0620 18:46:47.712745 (+ 235us) client_negotiation.cc:240] Negotiation successful
0620 18:46:47.713245 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":270,"threads_started":1}
I20260620 18:46:47.715891 25789 heartbeater.cc:347] Connected to a master server at 127.5.225.190:36747
I20260620 18:46:47.716202 25789 heartbeater.cc:464] Registering TS with master...
I20260620 18:46:47.716922 25789 heartbeater.cc:511] Master 127.5.225.190:36747 requested a full tablet report, sending...
I20260620 18:46:47.718542 25333 ts_manager.cc:194] Registered new tserver with Master: a3496e4323f941ae9e15dcb6782bbb09 (127.5.225.131:43335)
I20260620 18:46:47.719587 25333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.225.131@KRBTEST.COM'} at 127.5.225.131:58993
I20260620 18:46:47.719614 6022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:46:47.736498 25524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:47.724040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44204 (local address 127.5.225.190:36747)
0620 18:46:47.724210 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:47.724214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:47.724585 (+ 371us) server_negotiation.cc:407] Connection header received
0620 18:46:47.724714 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:47.724718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:47.724774 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:47.724854 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:47.726302 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:47.727018 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:47.728270 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:47.728701 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:47.728802 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:47.731223 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:47.731229 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:47.731234 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:46:47.731298 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:47.733922 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:47.734615 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:47.734619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:47.734621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:47.734744 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:47.735101 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:47.735104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:47.735106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:47.735519 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:47.735770 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:47.736052 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 18:46:47.736273 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 18:46:47.740710 25333 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:46:47.747546 25333 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:46:47.771387 25803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:47.760501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.131:43335 (local address 127.0.0.1:43468)
0620 18:46:47.762522 (+ 2021us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:47.762538 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:47.762659 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:47.763551 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:47.763560 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:47.763682 (+ 122us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:46:47.764293 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:47.764312 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:47.766049 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:47.766054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:47.768352 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:47.768366 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:47.770241 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:47.770248 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:47.770450 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:47.770509 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 18:46:47.770653 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 18:46:47.770870 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1905,"mutex_wait_us":192,"spinlock_wait_cycles":172160,"thread_start_us":1766,"threads_started":1}
I20260620 18:46:47.772148 25804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:47.761176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43468 (local address 127.5.225.131:43335)
0620 18:46:47.761970 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:47.761984 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:47.762644 (+ 660us) server_negotiation.cc:407] Connection header received
0620 18:46:47.762813 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:47.762820 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:47.763055 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:47.763251 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:46:47.764575 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:47.765904 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:47.768517 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:47.770006 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:47.770148 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:47.770420 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:47.771660 (+ 1240us) server_negotiation.cc:299] Negotiation successful
0620 18:46:47.771828 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":439,"threads_started":1}
I20260620 18:46:47.772259 25802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:47.760304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46740 (local address 127.5.225.129:40337)
0620 18:46:47.762302 (+ 1998us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:47.762313 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:47.762343 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:46:47.762459 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:47.762467 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:47.762952 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:47.763143 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:46:47.764250 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:47.765776 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:47.768676 (+ 2900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:47.770045 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:47.770190 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:47.770416 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:47.771861 (+ 1445us) server_negotiation.cc:299] Negotiation successful
0620 18:46:47.772010 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1801,"thread_start_us":1625,"threads_started":1}
I20260620 18:46:47.771785 25805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:47.763070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.130:33413 (local address 127.0.0.1:45472)
0620 18:46:47.764248 (+ 1178us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:47.764265 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:47.764372 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:47.765375 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:47.765381 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:47.765409 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:46:47.765724 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:47.765734 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:47.767616 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:47.767621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:47.769320 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:47.769332 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:47.770924 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:47.770930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:47.771115 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:47.771145 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 18:46:47.771223 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 18:46:47.771357 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1013,"thread_start_us":840,"threads_started":1}
I20260620 18:46:47.771560 25801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:47.760090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:40337 (local address 127.0.0.1:46740)
0620 18:46:47.761277 (+ 1187us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:47.761335 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:47.761565 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:47.763399 (+ 1834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:47.763410 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:47.763451 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:46:47.764000 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:47.764021 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:47.765920 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:47.765925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:47.768513 (+ 2588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:47.768528 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:47.770618 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:47.770624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:47.770811 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:47.770849 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 18:46:47.770938 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 18:46:47.771074 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":862,"mutex_wait_us":92,"spinlock_wait_cycles":3968,"thread_start_us":691,"threads_started":1}
I20260620 18:46:47.773703 25806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:47.763569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45472 (local address 127.5.225.130:33413)
0620 18:46:47.764830 (+ 1261us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:47.764839 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:47.764863 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:46:47.764959 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:47.764966 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:47.765165 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:47.765350 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:46:47.765889 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:47.767473 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:47.769481 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:47.770689 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:47.771625 (+ 936us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:47.771816 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:47.771878 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 18:46:47.772056 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1107,"thread_start_us":1010,"threads_started":1}
I20260620 18:46:47.776175 25457 tablet_service.cc:1467] Processing CreateTablet for tablet e685e6beca1f4c97bfa25cba6c8bf343 (DEFAULT_TABLE table=test-table [id=4c4e8ba6b63c41b8b410facccc6dab2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:46:47.776253 25724 tablet_service.cc:1467] Processing CreateTablet for tablet e685e6beca1f4c97bfa25cba6c8bf343 (DEFAULT_TABLE table=test-table [id=4c4e8ba6b63c41b8b410facccc6dab2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:46:47.776877 25591 tablet_service.cc:1467] Processing CreateTablet for tablet e685e6beca1f4c97bfa25cba6c8bf343 (DEFAULT_TABLE table=test-table [id=4c4e8ba6b63c41b8b410facccc6dab2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:46:47.777621 25457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e685e6beca1f4c97bfa25cba6c8bf343. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:46:47.777663 25724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e685e6beca1f4c97bfa25cba6c8bf343. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:46:47.778123 25591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e685e6beca1f4c97bfa25cba6c8bf343. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:46:47.788422 25807 tablet_bootstrap.cc:492] T e685e6beca1f4c97bfa25cba6c8bf343 P a503992b294745618bb796a526245cce: Bootstrap starting.
I20260620 18:46:47.791890 25808 tablet_bootstrap.cc:492] T e685e6beca1f4c97bfa25cba6c8bf343 P 316f0723e41445ddb5d3cd8e1e456eb0: Bootstrap starting.
I20260620 18:46:47.792834 25809 tablet_bootstrap.cc:492] T e685e6beca1f4c97bfa25cba6c8bf343 P a3496e4323f941ae9e15dcb6782bbb09: Bootstrap starting.
I20260620 18:46:47.793045 25807 tablet_bootstrap.cc:654] T e685e6beca1f4c97bfa25cba6c8bf343 P a503992b294745618bb796a526245cce: Neither blocks nor log segments found. Creating new log.
I20260620 18:46:47.794438 25807 log.cc:824] T e685e6beca1f4c97bfa25cba6c8bf343 P a503992b294745618bb796a526245cce: Log is configured to *not* fsync() on all Append() calls
I20260620 18:46:47.795336 25808 tablet_bootstrap.cc:654] T e685e6beca1f4c97bfa25cba6c8bf343 P 316f0723e41445ddb5d3cd8e1e456eb0: Neither blocks nor log segments found. Creating new log.
I20260620 18:46:47.795879 25809 tablet_bootstrap.cc:654] T e685e6beca1f4c97bfa25cba6c8bf343 P a3496e4323f941ae9e15dcb6782bbb09: Neither blocks nor log segments found. Creating new log.
I20260620 18:46:47.796497 25808 log.cc:824] T e685e6beca1f4c97bfa25cba6c8bf343 P 316f0723e41445ddb5d3cd8e1e456eb0: Log is configured to *not* fsync() on all Append() calls
I20260620 18:46:47.796844 25809 log.cc:824] T e685e6beca1f4c97bfa25cba6c8bf343 P a3496e4323f941ae9e15dcb6782bbb09: Log is configured to *not* fsync() on all Append() calls
I20260620 18:46:47.797706 25807 tablet_bootstrap.cc:492] T e685e6beca1f4c97bfa25cba6c8bf343 P a503992b294745618bb796a526245cce: No bootstrap required, opened a new log
I20260620 18:46:47.797976 25807 ts_tablet_manager.cc:1397] T e685e6beca1f4c97bfa25cba6c8bf343 P a503992b294745618bb796a526245cce: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260620 18:46:47.799815 25809 tablet_bootstrap.cc:492] T e685e6beca1f4c97bfa25cba6c8bf343 P a3496e4323f941ae9e15dcb6782bbb09: No bootstrap required, opened a new log
I20260620 18:46:47.800071 25809 ts_tablet_manager.cc:1397] T e685e6beca1f4c97bfa25cba6c8bf343 P a3496e4323f941ae9e15dcb6782bbb09: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 18:46:47.804643 25808 tablet_bootstrap.cc:492] T e685e6beca1f4c97bfa25cba6c8bf343 P 316f0723e41445ddb5d3cd8e1e456eb0: No bootstrap required, opened a new log
I20260620 18:46:47.804890 25808 ts_tablet_manager.cc:1397] T e685e6beca1f4c97bfa25cba6c8bf343 P 316f0723e41445ddb5d3cd8e1e456eb0: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260620 18:46:47.807612 25809 raft_consensus.cc:348] T e685e6beca1f4c97bfa25cba6c8bf343 P a3496e4323f941ae9e15dcb6782bbb09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a503992b294745618bb796a526245cce" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 40337 } } peers { permanent_uuid: "a3496e4323f941ae9e15dcb6782bbb09" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 43335 } } peers { permanent_uuid: "316f0723e41445ddb5d3cd8e1e456eb0" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33413 } }
I20260620 18:46:47.808036 25809 raft_consensus.cc:374] T e685e6beca1f4c97bfa25cba6c8bf343 P a3496e4323f941ae9e15dcb6782bbb09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:46:47.808161 25809 raft_consensus.cc:729] T e685e6beca1f4c97bfa25cba6c8bf343 P a3496e4323f941ae9e15dcb6782bbb09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3496e4323f941ae9e15dcb6782bbb09, State: Initialized, Role: FOLLOWER
I20260620 18:46:47.808908 25809 consensus_queue.cc:260] T e685e6beca1f4c97bfa25cba6c8bf343 P a3496e4323f941ae9e15dcb6782bbb09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a503992b294745618bb796a526245cce" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 40337 } } peers { permanent_uuid: "a3496e4323f941ae9e15dcb6782bbb09" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 43335 } } peers { permanent_uuid: "316f0723e41445ddb5d3cd8e1e456eb0" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33413 } }
I20260620 18:46:47.809330 25807 raft_consensus.cc:348] T e685e6beca1f4c97bfa25cba6c8bf343 P a503992b294745618bb796a526245cce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a503992b294745618bb796a526245cce" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 40337 } } peers { permanent_uuid: "a3496e4323f941ae9e15dcb6782bbb09" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 43335 } } peers { permanent_uuid: "316f0723e41445ddb5d3cd8e1e456eb0" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33413 } }
I20260620 18:46:47.809716 25807 raft_consensus.cc:374] T e685e6beca1f4c97bfa25cba6c8bf343 P a503992b294745618bb796a526245cce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:46:47.809799 25807 raft_consensus.cc:729] T e685e6beca1f4c97bfa25cba6c8bf343 P a503992b294745618bb796a526245cce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a503992b294745618bb796a526245cce, State: Initialized, Role: FOLLOWER
I20260620 18:46:47.810427 25807 consensus_queue.cc:260] T e685e6beca1f4c97bfa25cba6c8bf343 P a503992b294745618bb796a526245cce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a503992b294745618bb796a526245cce" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 40337 } } peers { permanent_uuid: "a3496e4323f941ae9e15dcb6782bbb09" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 43335 } } peers { permanent_uuid: "316f0723e41445ddb5d3cd8e1e456eb0" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33413 } }
I20260620 18:46:47.812150 25807 ts_tablet_manager.cc:1428] T e685e6beca1f4c97bfa25cba6c8bf343 P a503992b294745618bb796a526245cce: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260620 18:46:47.813405 25789 heartbeater.cc:503] Master 127.5.225.190:36747 was elected leader, sending a full tablet report...
I20260620 18:46:47.814818 25809 ts_tablet_manager.cc:1428] T e685e6beca1f4c97bfa25cba6c8bf343 P a3496e4323f941ae9e15dcb6782bbb09: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260620 18:46:47.816133 25808 raft_consensus.cc:348] T e685e6beca1f4c97bfa25cba6c8bf343 P 316f0723e41445ddb5d3cd8e1e456eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a503992b294745618bb796a526245cce" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 40337 } } peers { permanent_uuid: "a3496e4323f941ae9e15dcb6782bbb09" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 43335 } } peers { permanent_uuid: "316f0723e41445ddb5d3cd8e1e456eb0" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33413 } }
I20260620 18:46:47.816577 25808 raft_consensus.cc:374] T e685e6beca1f4c97bfa25cba6c8bf343 P 316f0723e41445ddb5d3cd8e1e456eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:46:47.816682 25808 raft_consensus.cc:729] T e685e6beca1f4c97bfa25cba6c8bf343 P 316f0723e41445ddb5d3cd8e1e456eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f0723e41445ddb5d3cd8e1e456eb0, State: Initialized, Role: FOLLOWER
I20260620 18:46:47.817282 25808 consensus_queue.cc:260] T e685e6beca1f4c97bfa25cba6c8bf343 P 316f0723e41445ddb5d3cd8e1e456eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a503992b294745618bb796a526245cce" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 40337 } } peers { permanent_uuid: "a3496e4323f941ae9e15dcb6782bbb09" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 43335 } } peers { permanent_uuid: "316f0723e41445ddb5d3cd8e1e456eb0" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33413 } }
I20260620 18:46:47.818733 25656 heartbeater.cc:503] Master 127.5.225.190:36747 was elected leader, sending a full tablet report...
I20260620 18:46:47.819242 25808 ts_tablet_manager.cc:1428] T e685e6beca1f4c97bfa25cba6c8bf343 P 316f0723e41445ddb5d3cd8e1e456eb0: Time spent starting tablet: real 0.014s user 0.008s sys 0.006s
I20260620 18:46:47.827080 25813 raft_consensus.cc:482] T e685e6beca1f4c97bfa25cba6c8bf343 P a3496e4323f941ae9e15dcb6782bbb09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:46:47.827312 25813 raft_consensus.cc:504] T e685e6beca1f4c97bfa25cba6c8bf343 P a3496e4323f941ae9e15dcb6782bbb09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a503992b294745618bb796a526245cce" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 40337 } } peers { permanent_uuid: "a3496e4323f941ae9e15dcb6782bbb09" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 43335 } } peers { permanent_uuid: "316f0723e41445ddb5d3cd8e1e456eb0" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33413 } }
I20260620 18:46:47.828900 25813 leader_election.cc:288] T e685e6beca1f4c97bfa25cba6c8bf343 P a3496e4323f941ae9e15dcb6782bbb09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a503992b294745618bb796a526245cce (127.5.225.129:40337), 316f0723e41445ddb5d3cd8e1e456eb0 (127.5.225.130:33413)
I20260620 18:46:47.836061 25791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:47.829169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:40337 (local address 127.5.225.131:53519)
0620 18:46:47.829493 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:47.829517 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:47.829649 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:47.830129 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:47.830186 (+ 57us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:47.830255 (+ 69us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:46:47.830711 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:47.830729 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:47.832092 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:47.832099 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:47.833682 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:47.833694 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:47.835259 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:47.835266 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:47.835457 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:47.835492 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 18:46:47.835593 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 18:46:47.835726 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170}
I20260620 18:46:47.836217 25802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:47.829343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:53519 (local address 127.5.225.129:40337)
0620 18:46:47.829568 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:47.829576 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:47.829598 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:46:47.829866 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:47.829872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:47.829966 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:47.830075 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:46:47.830877 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:47.831963 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:47.833848 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:47.835122 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:47.835241 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:47.835295 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:47.835661 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 18:46:47.835772 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260620 18:46:47.837284 25816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:47.829308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.130:33413 (local address 127.5.225.131:53981)
0620 18:46:47.830870 (+ 1562us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:47.830888 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:47.831015 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:47.831337 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:47.831341 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:47.831366 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:46:47.831705 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:47.831715 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:47.833027 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:47.833031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:47.834843 (+ 1812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:47.834858 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:47.836617 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:47.836622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:47.836740 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:47.836764 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 18:46:47.836837 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 18:46:47.836917 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1437,"mutex_wait_us":1,"thread_start_us":1202,"threads_started":1}
I20260620 18:46:47.837348 25806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:47.829441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:53981 (local address 127.5.225.130:33413)
0620 18:46:47.829616 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:47.829624 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:47.830978 (+ 1354us) server_negotiation.cc:407] Connection header received
0620 18:46:47.831132 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:47.831137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:47.831195 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:47.831302 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:46:47.831827 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:47.832867 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:47.834997 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:47.836294 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:47.836429 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:47.836477 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:47.836916 (+ 439us) server_negotiation.cc:299] Negotiation successful
0620 18:46:47.837072 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:46:47.837292 25477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e685e6beca1f4c97bfa25cba6c8bf343" candidate_uuid: "a3496e4323f941ae9e15dcb6782bbb09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a503992b294745618bb796a526245cce" is_pre_election: true
I20260620 18:46:47.837980 25477 raft_consensus.cc:2454] T e685e6beca1f4c97bfa25cba6c8bf343 P a503992b294745618bb796a526245cce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3496e4323f941ae9e15dcb6782bbb09 in term 0.
I20260620 18:46:47.838846 25677 leader_election.cc:302] T e685e6beca1f4c97bfa25cba6c8bf343 P a3496e4323f941ae9e15dcb6782bbb09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3496e4323f941ae9e15dcb6782bbb09, a503992b294745618bb796a526245cce; no voters:
I20260620 18:46:47.838917 25611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e685e6beca1f4c97bfa25cba6c8bf343" candidate_uuid: "a3496e4323f941ae9e15dcb6782bbb09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f0723e41445ddb5d3cd8e1e456eb0" is_pre_election: true
I20260620 18:46:47.839282 25813 raft_consensus.cc:2781] T e685e6beca1f4c97bfa25cba6c8bf343 P a3496e4323f941ae9e15dcb6782bbb09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:46:47.839452 25813 raft_consensus.cc:482] T e685e6beca1f4c97bfa25cba6c8bf343 P a3496e4323f941ae9e15dcb6782bbb09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:46:47.839494 25611 raft_consensus.cc:2454] T e685e6beca1f4c97bfa25cba6c8bf343 P 316f0723e41445ddb5d3cd8e1e456eb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3496e4323f941ae9e15dcb6782bbb09 in term 0.
I20260620 18:46:47.839584 25813 raft_consensus.cc:3037] T e685e6beca1f4c97bfa25cba6c8bf343 P a3496e4323f941ae9e15dcb6782bbb09 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:46:47.842041 25813 raft_consensus.cc:504] T e685e6beca1f4c97bfa25cba6c8bf343 P a3496e4323f941ae9e15dcb6782bbb09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a503992b294745618bb796a526245cce" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 40337 } } peers { permanent_uuid: "a3496e4323f941ae9e15dcb6782bbb09" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 43335 } } peers { permanent_uuid: "316f0723e41445ddb5d3cd8e1e456eb0" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33413 } }
I20260620 18:46:47.842775 25813 leader_election.cc:288] T e685e6beca1f4c97bfa25cba6c8bf343 P a3496e4323f941ae9e15dcb6782bbb09 [CANDIDATE]: Term 1 election: Requested vote from peers a503992b294745618bb796a526245cce (127.5.225.129:40337), 316f0723e41445ddb5d3cd8e1e456eb0 (127.5.225.130:33413)
I20260620 18:46:47.843232 25611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e685e6beca1f4c97bfa25cba6c8bf343" candidate_uuid: "a3496e4323f941ae9e15dcb6782bbb09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f0723e41445ddb5d3cd8e1e456eb0"
I20260620 18:46:47.843257 25477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e685e6beca1f4c97bfa25cba6c8bf343" candidate_uuid: "a3496e4323f941ae9e15dcb6782bbb09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a503992b294745618bb796a526245cce"
I20260620 18:46:47.843442 25611 raft_consensus.cc:3037] T e685e6beca1f4c97bfa25cba6c8bf343 P 316f0723e41445ddb5d3cd8e1e456eb0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:46:47.843467 25477 raft_consensus.cc:3037] T e685e6beca1f4c97bfa25cba6c8bf343 P a503992b294745618bb796a526245cce [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:46:47.845669 25611 raft_consensus.cc:2454] T e685e6beca1f4c97bfa25cba6c8bf343 P 316f0723e41445ddb5d3cd8e1e456eb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3496e4323f941ae9e15dcb6782bbb09 in term 1.
I20260620 18:46:47.845868 25477 raft_consensus.cc:2454] T e685e6beca1f4c97bfa25cba6c8bf343 P a503992b294745618bb796a526245cce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3496e4323f941ae9e15dcb6782bbb09 in term 1.
I20260620 18:46:47.846206 25678 leader_election.cc:302] T e685e6beca1f4c97bfa25cba6c8bf343 P a3496e4323f941ae9e15dcb6782bbb09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f0723e41445ddb5d3cd8e1e456eb0, a3496e4323f941ae9e15dcb6782bbb09; no voters:
I20260620 18:46:47.846580 25813 raft_consensus.cc:2781] T e685e6beca1f4c97bfa25cba6c8bf343 P a3496e4323f941ae9e15dcb6782bbb09 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:46:47.847040 25813 raft_consensus.cc:686] T e685e6beca1f4c97bfa25cba6c8bf343 P a3496e4323f941ae9e15dcb6782bbb09 [term 1 LEADER]: Becoming Leader. State: Replica: a3496e4323f941ae9e15dcb6782bbb09, State: Running, Role: LEADER
I20260620 18:46:47.847622 25813 consensus_queue.cc:237] T e685e6beca1f4c97bfa25cba6c8bf343 P a3496e4323f941ae9e15dcb6782bbb09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a503992b294745618bb796a526245cce" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 40337 } } peers { permanent_uuid: "a3496e4323f941ae9e15dcb6782bbb09" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 43335 } } peers { permanent_uuid: "316f0723e41445ddb5d3cd8e1e456eb0" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33413 } }
I20260620 18:46:47.852411 25333 catalog_manager.cc:5495] T e685e6beca1f4c97bfa25cba6c8bf343 P a3496e4323f941ae9e15dcb6782bbb09 reported cstate change: term changed from 0 to 1, leader changed from <none> to a3496e4323f941ae9e15dcb6782bbb09 (127.5.225.131). New cstate: current_term: 1 leader_uuid: "a3496e4323f941ae9e15dcb6782bbb09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a503992b294745618bb796a526245cce" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 40337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3496e4323f941ae9e15dcb6782bbb09" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 43335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "316f0723e41445ddb5d3cd8e1e456eb0" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33413 } health_report { overall_health: UNKNOWN } } }
I20260620 18:46:47.869325 25804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:47.864348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43474 (local address 127.5.225.131:43335)
0620 18:46:47.864539 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:47.864543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:47.864649 (+ 106us) server_negotiation.cc:407] Connection header received
0620 18:46:47.864825 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:47.864829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:47.864914 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:47.865049 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:46:47.865357 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:47.866094 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:47.867489 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:47.867896 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:47.868000 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:47.868402 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:46:47.868894 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:46:47.869026 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:47.869065 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 18:46:47.869128 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:46:47.877249 25477 raft_consensus.cc:1261] T e685e6beca1f4c97bfa25cba6c8bf343 P a503992b294745618bb796a526245cce [term 1 FOLLOWER]: Refusing update from remote peer a3496e4323f941ae9e15dcb6782bbb09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:46:47.877249 25611 raft_consensus.cc:1261] T e685e6beca1f4c97bfa25cba6c8bf343 P 316f0723e41445ddb5d3cd8e1e456eb0 [term 1 FOLLOWER]: Refusing update from remote peer a3496e4323f941ae9e15dcb6782bbb09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:46:47.878243 25817 consensus_queue.cc:1035] T e685e6beca1f4c97bfa25cba6c8bf343 P a3496e4323f941ae9e15dcb6782bbb09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a503992b294745618bb796a526245cce" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 40337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:46:47.878600 25813 consensus_queue.cc:1035] T e685e6beca1f4c97bfa25cba6c8bf343 P a3496e4323f941ae9e15dcb6782bbb09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f0723e41445ddb5d3cd8e1e456eb0" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 33413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:46:47.888072 25820 mvcc.cc:205] Tried to move back new op lower bound from 7298995027455684608 to 7298995027347939328. Current Snapshot: MvccSnapshot[applied={T|T < 7298995027455684608}]
I20260620 18:46:47.901566 25332 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:46:47.901899 25332 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:46:47.905930 25332 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 273b3f828ea84638a353c6aa059704fa: Sending DeleteTablet for 3 replicas of tablet e685e6beca1f4c97bfa25cba6c8bf343
I20260620 18:46:47.907147 25457 tablet_service.cc:1514] Processing DeleteTablet for tablet e685e6beca1f4c97bfa25cba6c8bf343 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:46:47 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:46740
I20260620 18:46:47.907424 25591 tablet_service.cc:1514] Processing DeleteTablet for tablet e685e6beca1f4c97bfa25cba6c8bf343 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:46:47 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:45472
I20260620 18:46:47.907724 25724 tablet_service.cc:1514] Processing DeleteTablet for tablet e685e6beca1f4c97bfa25cba6c8bf343 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:46:47 UTC) from {username='kudu', principal='kudu/127.5.225.190@KRBTEST.COM'} at 127.0.0.1:43468
I20260620 18:46:47.909260 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 25394
I20260620 18:46:47.917795 25831 tablet_replica.cc:330] T e685e6beca1f4c97bfa25cba6c8bf343 P a3496e4323f941ae9e15dcb6782bbb09: stopping tablet replica
I20260620 18:46:47.918885 25831 raft_consensus.cc:2229] T e685e6beca1f4c97bfa25cba6c8bf343 P a3496e4323f941ae9e15dcb6782bbb09 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:46:47.919687 25831 raft_consensus.cc:2258] T e685e6beca1f4c97bfa25cba6c8bf343 P a3496e4323f941ae9e15dcb6782bbb09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:46:47.920488 25829 tablet_replica.cc:330] T e685e6beca1f4c97bfa25cba6c8bf343 P a503992b294745618bb796a526245cce: stopping tablet replica
I20260620 18:46:47.920979 25829 raft_consensus.cc:2229] T e685e6beca1f4c97bfa25cba6c8bf343 P a503992b294745618bb796a526245cce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:46:47.921535 25829 raft_consensus.cc:2258] T e685e6beca1f4c97bfa25cba6c8bf343 P a503992b294745618bb796a526245cce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:46:47.923034 25830 tablet_replica.cc:330] T e685e6beca1f4c97bfa25cba6c8bf343 P 316f0723e41445ddb5d3cd8e1e456eb0: stopping tablet replica
I20260620 18:46:47.923436 25830 raft_consensus.cc:2229] T e685e6beca1f4c97bfa25cba6c8bf343 P 316f0723e41445ddb5d3cd8e1e456eb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:46:47.923926 25830 raft_consensus.cc:2258] T e685e6beca1f4c97bfa25cba6c8bf343 P 316f0723e41445ddb5d3cd8e1e456eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:46:47.924091 25829 ts_tablet_manager.cc:1905] T e685e6beca1f4c97bfa25cba6c8bf343 P a503992b294745618bb796a526245cce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:46:47.924109 25831 ts_tablet_manager.cc:1905] T e685e6beca1f4c97bfa25cba6c8bf343 P a3496e4323f941ae9e15dcb6782bbb09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:46:47.925927 25830 ts_tablet_manager.cc:1905] T e685e6beca1f4c97bfa25cba6c8bf343 P 316f0723e41445ddb5d3cd8e1e456eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 18:46:47 dist-test-slave-k6lg krb5kdc[25288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.129@KRBTEST.COM
I20260620 18:46:47.932230 25831 ts_tablet_manager.cc:1918] T e685e6beca1f4c97bfa25cba6c8bf343 P a3496e4323f941ae9e15dcb6782bbb09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:46:47.932417 25831 log.cc:1196] T e685e6beca1f4c97bfa25cba6c8bf343 P a3496e4323f941ae9e15dcb6782bbb09: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-2/wal/wals/e685e6beca1f4c97bfa25cba6c8bf343
I20260620 18:46:47.933023 25831 ts_tablet_manager.cc:1939] T e685e6beca1f4c97bfa25cba6c8bf343 P a3496e4323f941ae9e15dcb6782bbb09: Deleting consensus metadata
I20260620 18:46:47.933063 25830 ts_tablet_manager.cc:1918] T e685e6beca1f4c97bfa25cba6c8bf343 P 316f0723e41445ddb5d3cd8e1e456eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:46:47.933341 25830 log.cc:1196] T e685e6beca1f4c97bfa25cba6c8bf343 P 316f0723e41445ddb5d3cd8e1e456eb0: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-1/wal/wals/e685e6beca1f4c97bfa25cba6c8bf343
I20260620 18:46:47.933902 25830 ts_tablet_manager.cc:1939] T e685e6beca1f4c97bfa25cba6c8bf343 P 316f0723e41445ddb5d3cd8e1e456eb0: Deleting consensus metadata
I20260620 18:46:47.935062 25320 catalog_manager.cc:4841] TS a3496e4323f941ae9e15dcb6782bbb09 (127.5.225.131:43335): tablet e685e6beca1f4c97bfa25cba6c8bf343 (table test-table [id=4c4e8ba6b63c41b8b410facccc6dab2a]) successfully deleted
I20260620 18:46:47.936573 25318 catalog_manager.cc:4841] TS 316f0723e41445ddb5d3cd8e1e456eb0 (127.5.225.130:33413): tablet e685e6beca1f4c97bfa25cba6c8bf343 (table test-table [id=4c4e8ba6b63c41b8b410facccc6dab2a]) successfully deleted
I20260620 18:46:47.939340 25829 ts_tablet_manager.cc:1918] T e685e6beca1f4c97bfa25cba6c8bf343 P a503992b294745618bb796a526245cce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:46:47.939594 25829 log.cc:1196] T e685e6beca1f4c97bfa25cba6c8bf343 P a503992b294745618bb796a526245cce: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980894474392-6022-0/minicluster-data/ts-0/wal/wals/e685e6beca1f4c97bfa25cba6c8bf343
I20260620 18:46:47.940061 25802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:47.910326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46744 (local address 127.5.225.129:40337)
0620 18:46:47.910513 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:47.910519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:47.918675 (+ 8156us) server_negotiation.cc:407] Connection header received
0620 18:46:47.919427 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:47.919433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:47.919532 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:47.919613 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:47.921597 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:47.922813 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:47.926155 (+ 3342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:47.926849 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:47.926946 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:47.932404 (+ 5458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:47.932421 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:47.932439 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:46:47.932491 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:47.936438 (+ 3947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:47.937385 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:47.937395 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:47.937398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:47.937634 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:47.938085 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:47.938090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:47.938094 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:47.938641 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:47.938992 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:47.939305 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 18:46:47.939559 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:46:47.940464 25829 ts_tablet_manager.cc:1939] T e685e6beca1f4c97bfa25cba6c8bf343 P a503992b294745618bb796a526245cce: Deleting consensus metadata
I20260620 18:46:47.940762 25517 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:46:48.032080 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 25394
I20260620 18:46:48.032215 25317 catalog_manager.cc:4841] TS a503992b294745618bb796a526245cce (127.5.225.129:40337): tablet e685e6beca1f4c97bfa25cba6c8bf343 (table test-table [id=4c4e8ba6b63c41b8b410facccc6dab2a]) successfully deleted
I20260620 18:46:48.050539 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 25528
Jun 20 18:46:48 dist-test-slave-k6lg krb5kdc[25288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.130@KRBTEST.COM
I20260620 18:46:48.067975 25806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:48.051417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45474 (local address 127.5.225.130:33413)
0620 18:46:48.051585 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:48.051589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:48.051606 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:46:48.051844 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:48.051848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:48.051917 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:48.052062 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:48.053674 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:48.054496 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:48.056303 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:48.056681 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:48.056771 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:48.061831 (+ 5060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:48.061845 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:48.061861 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:46:48.061895 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:48.064800 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:48.065573 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:48.065581 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:48.065583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:48.065707 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:48.066093 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:48.066101 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:48.066105 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:48.066701 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:48.067002 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:48.067301 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 18:46:48.067517 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 18:46:48.068717 25651 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:46:48.160815 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 25528
I20260620 18:46:48.178443 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 25661
Jun 20 18:46:48 dist-test-slave-k6lg krb5kdc[25288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.131@KRBTEST.COM
I20260620 18:46:48.194947 25804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:48.179277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43478 (local address 127.5.225.131:43335)
0620 18:46:48.179449 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:48.179456 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:48.179540 (+ 84us) server_negotiation.cc:407] Connection header received
0620 18:46:48.179760 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:48.179766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:48.179869 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:48.179982 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:48.181295 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:48.182097 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:48.183534 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:48.183996 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:48.184176 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:48.188652 (+ 4476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:48.188667 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:48.188682 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:46:48.188717 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:48.192059 (+ 3342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:48.192791 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:48.192799 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:48.192801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:48.192958 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:48.193346 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:48.193352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:48.193355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:48.193719 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:48.194065 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:48.194349 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 18:46:48.194599 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:46:48.195680 25784 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:46:48.319248 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 25661
I20260620 18:46:48.337082 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 25301
I20260620 18:46:48.338230 25363 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:46:48.448123 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 25301
2026-06-20T18:46:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4177 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T18:46:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:46:48Z Disabled control of system clock
I20260620 18:46:48.522702 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:42779
--webserver_interface=127.5.225.190
--webserver_port=0
--builtin_ntp_servers=127.5.225.148:43767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:42779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:46:48.850713 25840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:46:48.851037 25840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:46:48.851125 25840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:46:48.858633 25840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:46:48.858753 25840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:46:48.858793 25840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:46:48.858842 25840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:46:48.858887 25840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:46:48.868438 25840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:43767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:42779
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:42779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:46:48.869694 25840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:46:48.871480 25840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:46:48.883045 25845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:48.883020 25848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:48.882965 25846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:46:48.883751 25840 server_base.cc:1034] running on GCE node
I20260620 18:46:48.884477 25840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:46:48.885782 25840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:46:48.886981 25840 hybrid_clock.cc:648] HybridClock initialized: now 1781981208886930 us; error 49 us; skew 500 ppm
I20260620 18:46:48.887347 25840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:46:48.889609 25840 webserver.cc:460] Webserver started at http://127.5.225.190:46643/ using document root <none> and password file <none>
I20260620 18:46:48.890388 25840 fs_manager.cc:359] Metadata directory not provided
I20260620 18:46:48.890506 25840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:46:48.890832 25840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:46:48.893651 25840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "110a6e6b7db0488f8b448ce28bbcffbd"
format_stamp: "Formatted at 2026-06-20 18:46:48 on dist-test-slave-k6lg"
server_key: "33b4f82978c509909098058512c3b1de"
server_key_iv: "bf2f3c9901013adfa531fdf280099572"
server_key_version: "encryptionkey@0"
I20260620 18:46:48.894481 25840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "110a6e6b7db0488f8b448ce28bbcffbd"
format_stamp: "Formatted at 2026-06-20 18:46:48 on dist-test-slave-k6lg"
server_key: "33b4f82978c509909098058512c3b1de"
server_key_iv: "bf2f3c9901013adfa531fdf280099572"
server_key_version: "encryptionkey@0"
I20260620 18:46:48.899422 25840 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 18:46:48.902822 25854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:48.904138 25840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:46:48.904327 25840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "110a6e6b7db0488f8b448ce28bbcffbd"
format_stamp: "Formatted at 2026-06-20 18:46:48 on dist-test-slave-k6lg"
server_key: "33b4f82978c509909098058512c3b1de"
server_key_iv: "bf2f3c9901013adfa531fdf280099572"
server_key_version: "encryptionkey@0"
I20260620 18:46:48.904546 25840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:46:48.941015 25840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:46:48.941913 25840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:46:48.942247 25840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:46:48.960255 25840 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:42779
I20260620 18:46:48.960275 25905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:42779 every 8 connection(s)
I20260620 18:46:48.961659 25840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:46:48.965306 25906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:46:48.965981 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 25840
I20260620 18:46:48.966218 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:46:48.966771 6022 external_mini_cluster.cc:1428] Setting key 199ed20352ef23babab22faf38e99bf4
I20260620 18:46:48.978655 25906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 110a6e6b7db0488f8b448ce28bbcffbd: Bootstrap starting.
I20260620 18:46:48.980132 25909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:48.968719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54884 (local address 127.5.225.190:42779)
0620 18:46:48.969705 (+ 986us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:48.969715 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:48.969761 (+ 46us) server_negotiation.cc:407] Connection header received
0620 18:46:48.971456 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:48.971477 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:48.971932 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:48.972249 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:48.972808 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:48.974025 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:48.975896 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:48.976574 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:48.976696 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:48.977857 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:48.977864 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:48.977877 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:46:48.978446 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:48.978544 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:46:48.978565 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:48.978653 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:48.978919 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 18:46:48.979242 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":368,"threads_started":1}
I20260620 18:46:48.981765 25906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 110a6e6b7db0488f8b448ce28bbcffbd: Neither blocks nor log segments found. Creating new log.
I20260620 18:46:48.982805 25906 log.cc:824] T 00000000000000000000000000000000 P 110a6e6b7db0488f8b448ce28bbcffbd: Log is configured to *not* fsync() on all Append() calls
I20260620 18:46:48.985212 25906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 110a6e6b7db0488f8b448ce28bbcffbd: No bootstrap required, opened a new log
I20260620 18:46:48.992218 25906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 110a6e6b7db0488f8b448ce28bbcffbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110a6e6b7db0488f8b448ce28bbcffbd" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 42779 } }
I20260620 18:46:48.992550 25906 raft_consensus.cc:374] T 00000000000000000000000000000000 P 110a6e6b7db0488f8b448ce28bbcffbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:46:48.992673 25906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 110a6e6b7db0488f8b448ce28bbcffbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 110a6e6b7db0488f8b448ce28bbcffbd, State: Initialized, Role: FOLLOWER
I20260620 18:46:48.993183 25906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 110a6e6b7db0488f8b448ce28bbcffbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110a6e6b7db0488f8b448ce28bbcffbd" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 42779 } }
I20260620 18:46:48.993408 25906 raft_consensus.cc:388] T 00000000000000000000000000000000 P 110a6e6b7db0488f8b448ce28bbcffbd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:46:48.993563 25906 raft_consensus.cc:482] T 00000000000000000000000000000000 P 110a6e6b7db0488f8b448ce28bbcffbd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:46:48.993736 25906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 110a6e6b7db0488f8b448ce28bbcffbd [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:46:48.995854 25906 raft_consensus.cc:504] T 00000000000000000000000000000000 P 110a6e6b7db0488f8b448ce28bbcffbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110a6e6b7db0488f8b448ce28bbcffbd" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 42779 } }
I20260620 18:46:48.996313 25906 leader_election.cc:302] T 00000000000000000000000000000000 P 110a6e6b7db0488f8b448ce28bbcffbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 110a6e6b7db0488f8b448ce28bbcffbd; no voters:
I20260620 18:46:48.996922 25906 leader_election.cc:288] T 00000000000000000000000000000000 P 110a6e6b7db0488f8b448ce28bbcffbd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:46:48.997076 25911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 110a6e6b7db0488f8b448ce28bbcffbd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:46:48.997838 25911 raft_consensus.cc:686] T 00000000000000000000000000000000 P 110a6e6b7db0488f8b448ce28bbcffbd [term 1 LEADER]: Becoming Leader. State: Replica: 110a6e6b7db0488f8b448ce28bbcffbd, State: Running, Role: LEADER
I20260620 18:46:48.998366 25911 consensus_queue.cc:237] T 00000000000000000000000000000000 P 110a6e6b7db0488f8b448ce28bbcffbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110a6e6b7db0488f8b448ce28bbcffbd" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 42779 } }
I20260620 18:46:48.998988 25906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 110a6e6b7db0488f8b448ce28bbcffbd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:46:49.001034 25913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 110a6e6b7db0488f8b448ce28bbcffbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 110a6e6b7db0488f8b448ce28bbcffbd. Latest consensus state: current_term: 1 leader_uuid: "110a6e6b7db0488f8b448ce28bbcffbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110a6e6b7db0488f8b448ce28bbcffbd" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 42779 } } }
I20260620 18:46:49.001421 25913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 110a6e6b7db0488f8b448ce28bbcffbd [sys.catalog]: This master's current role is: LEADER
I20260620 18:46:49.000964 25912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 110a6e6b7db0488f8b448ce28bbcffbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "110a6e6b7db0488f8b448ce28bbcffbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110a6e6b7db0488f8b448ce28bbcffbd" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 42779 } } }
I20260620 18:46:49.001857 25912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 110a6e6b7db0488f8b448ce28bbcffbd [sys.catalog]: This master's current role is: LEADER
I20260620 18:46:49.003542 25918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:46:49.008944 25918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:46:49.018018 25918 catalog_manager.cc:1347] Generated new cluster ID: a15d76aea5a8462a82b4acc8957d5cf5
I20260620 18:46:49.018216 25918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:46:49.027760 25918 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:46:49.028934 25918 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:46:49.039505 25918 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 110a6e6b7db0488f8b448ce28bbcffbd: Generated new TSK 0
I20260620 18:46:49.040294 25918 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:46:49.049836 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.129:0
--local_ip_for_outbound_sockets=127.5.225.129
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:42779
--builtin_ntp_servers=127.5.225.148:43767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:46:49.376431 25930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:46:49.376699 25930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:46:49.376782 25930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:46:49.383708 25930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:46:49.383795 25930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:46:49.383898 25930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.129
I20260620 18:46:49.392549 25930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:43767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.225.129
--webserver_port=0
--tserver_master_addrs=127.5.225.190:42779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.129
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:46:49.393704 25930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:46:49.395956 25930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:46:49.414593 25935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:49.414655 25936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:49.414624 25938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:46:49.415309 25930 server_base.cc:1034] running on GCE node
I20260620 18:46:49.415867 25930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:46:49.416729 25930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:46:49.417923 25930 hybrid_clock.cc:648] HybridClock initialized: now 1781981209417866 us; error 59 us; skew 500 ppm
I20260620 18:46:49.418304 25930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:46:49.420408 25930 webserver.cc:460] Webserver started at http://127.5.225.129:41901/ using document root <none> and password file <none>
I20260620 18:46:49.421083 25930 fs_manager.cc:359] Metadata directory not provided
I20260620 18:46:49.421198 25930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:46:49.421518 25930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:46:49.424283 25930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-0/data/instance:
uuid: "dbe157a831e24528b98dcc50a321c00e"
format_stamp: "Formatted at 2026-06-20 18:46:49 on dist-test-slave-k6lg"
server_key: "88e0c23ff81afcdbf601b907651ee1f5"
server_key_iv: "ca5e970c2d4ec0e37ed0959298086f5a"
server_key_version: "encryptionkey@0"
I20260620 18:46:49.425063 25930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance:
uuid: "dbe157a831e24528b98dcc50a321c00e"
format_stamp: "Formatted at 2026-06-20 18:46:49 on dist-test-slave-k6lg"
server_key: "88e0c23ff81afcdbf601b907651ee1f5"
server_key_iv: "ca5e970c2d4ec0e37ed0959298086f5a"
server_key_version: "encryptionkey@0"
I20260620 18:46:49.429843 25930 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260620 18:46:49.433256 25944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:49.434562 25930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 18:46:49.434741 25930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-0/wal
uuid: "dbe157a831e24528b98dcc50a321c00e"
format_stamp: "Formatted at 2026-06-20 18:46:49 on dist-test-slave-k6lg"
server_key: "88e0c23ff81afcdbf601b907651ee1f5"
server_key_iv: "ca5e970c2d4ec0e37ed0959298086f5a"
server_key_version: "encryptionkey@0"
I20260620 18:46:49.434990 25930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:46:49.463456 25930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:46:49.464274 25930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:46:49.464576 25930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:46:49.465528 25930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:46:49.467033 25930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:46:49.467157 25930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:49.467296 25930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:46:49.467388 25930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:49.500106 25930 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.129:39057
I20260620 18:46:49.500192 26056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.129:39057 every 8 connection(s)
I20260620 18:46:49.501434 25930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-0/data/info.pb
I20260620 18:46:49.507148 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 25930
I20260620 18:46:49.507305 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-0/wal/instance
I20260620 18:46:49.507802 6022 external_mini_cluster.cc:1428] Setting key a2cae815d230d6f1dc2b932d4f34cbdf
I20260620 18:46:49.510746 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.130:0
--local_ip_for_outbound_sockets=127.5.225.130
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:42779
--builtin_ntp_servers=127.5.225.148:43767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:46:49.515738 26059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:49.504361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.129:55497 (local address 127.5.225.190:42779)
0620 18:46:49.504836 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:49.504841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:49.506027 (+ 1186us) server_negotiation.cc:407] Connection header received
0620 18:46:49.508160 (+ 2133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:49.508166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:49.508256 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:49.508386 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:49.509621 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:49.510461 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:49.512949 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:49.513359 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:49.513456 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:49.515017 (+ 1561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:49.515021 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:49.515025 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:46:49.515041 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:49.515080 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:46:49.515092 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:49.515163 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:49.515412 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 18:46:49.515489 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":269,"threads_started":1}
I20260620 18:46:49.517217 26060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:49.504995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:42779 (local address 127.5.225.129:55497)
0620 18:46:49.505869 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:49.505910 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:49.507884 (+ 1974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:49.508648 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:49.508661 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:49.508773 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:46:49.509440 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:49.509468 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:49.510625 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:49.510631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:49.512812 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:49.512829 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:49.513578 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:49.513587 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:49.513904 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:49.514668 (+ 764us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:46:49.514674 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:46:49.514730 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:46:49.514828 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:46:49.515191 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:46:49.515204 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:46:49.515215 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 18:46:49.515402 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 18:46:49.515836 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":259,"threads_started":1}
I20260620 18:46:49.518740 26057 heartbeater.cc:347] Connected to a master server at 127.5.225.190:42779
I20260620 18:46:49.519032 26057 heartbeater.cc:464] Registering TS with master...
I20260620 18:46:49.519903 26057 heartbeater.cc:511] Master 127.5.225.190:42779 requested a full tablet report, sending...
I20260620 18:46:49.522290 25871 ts_manager.cc:194] Registered new tserver with Master: dbe157a831e24528b98dcc50a321c00e (127.5.225.129:39057)
I20260620 18:46:49.524480 25871 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.225.129:55497
W20260620 18:46:49.846293 26061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:46:49.846580 26061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:46:49.846665 26061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:46:49.853717 26061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:46:49.853816 26061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:46:49.853919 26061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.130
I20260620 18:46:49.862816 26061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:43767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.225.130
--webserver_port=0
--tserver_master_addrs=127.5.225.190:42779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.130
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:46:49.864313 26061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:46:49.866004 26061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:46:49.879047 26069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:49.879153 26066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:49.879138 26067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:46:49.879829 26061 server_base.cc:1034] running on GCE node
I20260620 18:46:49.880524 26061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:46:49.881445 26061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:46:49.882647 26061 hybrid_clock.cc:648] HybridClock initialized: now 1781981209882603 us; error 31 us; skew 500 ppm
I20260620 18:46:49.882917 26061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:46:49.885167 26061 webserver.cc:460] Webserver started at http://127.5.225.130:43343/ using document root <none> and password file <none>
I20260620 18:46:49.885799 26061 fs_manager.cc:359] Metadata directory not provided
I20260620 18:46:49.885885 26061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:46:49.886137 26061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:46:49.888963 26061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-1/data/instance:
uuid: "4a79ba5b7dd146edbdaea8b3e7a7ec7a"
format_stamp: "Formatted at 2026-06-20 18:46:49 on dist-test-slave-k6lg"
server_key: "9f996212608ef7fa3df06669ef17fb55"
server_key_iv: "677331a9dac71e9d48f419484dec6c36"
server_key_version: "encryptionkey@0"
I20260620 18:46:49.889667 26061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance:
uuid: "4a79ba5b7dd146edbdaea8b3e7a7ec7a"
format_stamp: "Formatted at 2026-06-20 18:46:49 on dist-test-slave-k6lg"
server_key: "9f996212608ef7fa3df06669ef17fb55"
server_key_iv: "677331a9dac71e9d48f419484dec6c36"
server_key_version: "encryptionkey@0"
I20260620 18:46:49.894613 26061 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260620 18:46:49.897787 26075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:49.899211 26061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:46:49.899343 26061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-1/wal
uuid: "4a79ba5b7dd146edbdaea8b3e7a7ec7a"
format_stamp: "Formatted at 2026-06-20 18:46:49 on dist-test-slave-k6lg"
server_key: "9f996212608ef7fa3df06669ef17fb55"
server_key_iv: "677331a9dac71e9d48f419484dec6c36"
server_key_version: "encryptionkey@0"
I20260620 18:46:49.899526 26061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:46:49.914647 26061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:46:49.915427 26061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:46:49.915735 26061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:46:49.916669 26061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:46:49.918054 26061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:46:49.918164 26061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:49.918322 26061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:46:49.918421 26061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:49.951262 26061 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.130:44333
I20260620 18:46:49.951310 26187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.130:44333 every 8 connection(s)
I20260620 18:46:49.952585 26061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-1/data/info.pb
I20260620 18:46:49.959353 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 26061
I20260620 18:46:49.959545 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-1/wal/instance
I20260620 18:46:49.960091 6022 external_mini_cluster.cc:1428] Setting key b5b348384aa4ddd017da4c43c53dd17f
I20260620 18:46:49.964593 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.225.131:0
--local_ip_for_outbound_sockets=127.5.225.131
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:42779
--builtin_ntp_servers=127.5.225.148:43767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:46:49.967090 26059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:49.955138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:53083 (local address 127.5.225.190:42779)
0620 18:46:49.955365 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:49.955371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:49.956710 (+ 1339us) server_negotiation.cc:407] Connection header received
0620 18:46:49.958477 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:49.958481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:49.958553 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:49.958643 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:49.960056 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:49.961263 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:49.963722 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:49.964376 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:49.964516 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:49.966084 (+ 1568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:49.966093 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:49.966102 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:46:49.966127 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:49.966193 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:46:49.966212 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:49.966313 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:49.966547 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 18:46:49.966661 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 18:46:49.968122 26190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:49.955690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:42779 (local address 127.5.225.130:53083)
0620 18:46:49.956537 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:49.956572 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:49.958154 (+ 1582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:49.958910 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:49.958924 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:49.959038 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:46:49.959866 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:49.959893 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:49.961469 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:49.961477 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:49.963563 (+ 2086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:49.963581 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:49.964602 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:49.964610 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:49.964880 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:49.965704 (+ 824us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:46:49.965708 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:46:49.965761 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:46:49.965868 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:46:49.966321 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:46:49.966331 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:46:49.966339 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 18:46:49.966512 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 18:46:49.966899 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":250,"threads_started":1}
I20260620 18:46:49.969620 26188 heartbeater.cc:347] Connected to a master server at 127.5.225.190:42779
I20260620 18:46:49.969973 26188 heartbeater.cc:464] Registering TS with master...
I20260620 18:46:49.970945 26188 heartbeater.cc:511] Master 127.5.225.190:42779 requested a full tablet report, sending...
I20260620 18:46:49.972549 25871 ts_manager.cc:194] Registered new tserver with Master: 4a79ba5b7dd146edbdaea8b3e7a7ec7a (127.5.225.130:44333)
I20260620 18:46:49.973680 25871 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.225.130:53083
W20260620 18:46:50.301746 26191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:46:50.302047 26191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:46:50.302259 26191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:46:50.310375 26191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:46:50.310491 26191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:46:50.310681 26191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.225.131
I20260620 18:46:50.319618 26191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:43767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.225.131
--webserver_port=0
--tserver_master_addrs=127.5.225.190:42779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.225.131
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:46:50.320858 26191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:46:50.322649 26191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:46:50.335389 26199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:50.335363 26197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:50.335482 26196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:46:50.336170 26191 server_base.cc:1034] running on GCE node
I20260620 18:46:50.336786 26191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:46:50.337713 26191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:46:50.338941 26191 hybrid_clock.cc:648] HybridClock initialized: now 1781981210338901 us; error 36 us; skew 500 ppm
I20260620 18:46:50.339272 26191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:46:50.341565 26191 webserver.cc:460] Webserver started at http://127.5.225.131:40185/ using document root <none> and password file <none>
I20260620 18:46:50.342233 26191 fs_manager.cc:359] Metadata directory not provided
I20260620 18:46:50.342326 26191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:46:50.342600 26191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:46:50.345268 26191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-2/data/instance:
uuid: "920d1a3abe914aa1ae7d22b643af4cd9"
format_stamp: "Formatted at 2026-06-20 18:46:50 on dist-test-slave-k6lg"
server_key: "61ce01f8eef59208f25549bfcb9fcf3a"
server_key_iv: "4cc21d08af0e120a01bc204f135fd203"
server_key_version: "encryptionkey@0"
I20260620 18:46:50.345966 26191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance:
uuid: "920d1a3abe914aa1ae7d22b643af4cd9"
format_stamp: "Formatted at 2026-06-20 18:46:50 on dist-test-slave-k6lg"
server_key: "61ce01f8eef59208f25549bfcb9fcf3a"
server_key_iv: "4cc21d08af0e120a01bc204f135fd203"
server_key_version: "encryptionkey@0"
I20260620 18:46:50.350880 26191 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 18:46:50.354336 26205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:50.355608 26191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:46:50.355767 26191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-2/wal
uuid: "920d1a3abe914aa1ae7d22b643af4cd9"
format_stamp: "Formatted at 2026-06-20 18:46:50 on dist-test-slave-k6lg"
server_key: "61ce01f8eef59208f25549bfcb9fcf3a"
server_key_iv: "4cc21d08af0e120a01bc204f135fd203"
server_key_version: "encryptionkey@0"
I20260620 18:46:50.355969 26191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:46:50.377300 26191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:46:50.378060 26191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:46:50.378367 26191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:46:50.379289 26191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:46:50.380678 26191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:46:50.380795 26191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:50.380914 26191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:46:50.380998 26191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:50.413344 26191 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.131:40617
I20260620 18:46:50.413430 26317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.131:40617 every 8 connection(s)
I20260620 18:46:50.414731 26191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-2/data/info.pb
I20260620 18:46:50.422708 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 26191
I20260620 18:46:50.422891 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-2/wal/instance
I20260620 18:46:50.423347 6022 external_mini_cluster.cc:1428] Setting key 4be42bd2c4dfb822d87f6395e1b5e510
I20260620 18:46:50.427078 26059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:50.417289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.131:50873 (local address 127.5.225.190:42779)
0620 18:46:50.417505 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:50.417511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:50.418899 (+ 1388us) server_negotiation.cc:407] Connection header received
0620 18:46:50.420612 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:50.420616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:50.420682 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:50.420768 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:50.421766 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:50.422706 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:50.424369 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:50.424789 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:50.424924 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:50.426280 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:50.426285 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:50.426288 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:46:50.426305 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:50.426338 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:46:50.426350 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:50.426446 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:50.426659 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 18:46:50.426756 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 18:46:50.428062 26320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:50.417797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:42779 (local address 127.5.225.131:50873)
0620 18:46:50.418741 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:50.418783 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:50.420365 (+ 1582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:50.420958 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:50.420966 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:50.421088 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:46:50.421630 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:50.421647 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:50.422892 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:50.422896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:50.424213 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:50.424225 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:50.425014 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:50.425021 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:50.425243 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:50.425933 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:46:50.425936 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:46:50.425975 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:46:50.426062 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:46:50.426446 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:46:50.426459 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:46:50.426469 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 18:46:50.426650 (+ 181us) client_negotiation.cc:240] Negotiation successful
0620 18:46:50.427033 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":336,"threads_started":1}
I20260620 18:46:50.429307 26318 heartbeater.cc:347] Connected to a master server at 127.5.225.190:42779
I20260620 18:46:50.429606 26318 heartbeater.cc:464] Registering TS with master...
I20260620 18:46:50.430382 26318 heartbeater.cc:511] Master 127.5.225.190:42779 requested a full tablet report, sending...
I20260620 18:46:50.431743 25871 ts_manager.cc:194] Registered new tserver with Master: 920d1a3abe914aa1ae7d22b643af4cd9 (127.5.225.131:40617)
I20260620 18:46:50.432541 25871 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.225.131:50873
I20260620 18:46:50.438205 6022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:46:50.448119 26059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:50.442150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54894 (local address 127.5.225.190:42779)
0620 18:46:50.442344 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:50.442348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:50.442796 (+ 448us) server_negotiation.cc:407] Connection header received
0620 18:46:50.442994 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:50.443000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:50.443090 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:50.443194 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:50.443734 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:50.444421 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:50.445792 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:50.446265 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:50.446398 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:50.447447 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:50.447452 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:50.447456 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:46:50.447473 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:50.447507 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:46:50.447519 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:50.447589 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:50.447705 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 18:46:50.447780 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 18:46:50.451773 25871 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54894:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:46:50.458730 25871 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:46:50.493607 26331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:50.470734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:39057 (local address 127.0.0.1:51762)
0620 18:46:50.482653 (+ 11919us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:50.482679 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:50.482857 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:50.484292 (+ 1435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:50.484302 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:50.484345 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:46:50.484829 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:50.484845 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:50.488491 (+ 3646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:50.488499 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:50.491077 (+ 2578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:50.491095 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:50.492494 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:50.492500 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:50.492794 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:50.492853 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 18:46:50.493012 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 18:46:50.493221 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11739,"mutex_wait_us":49,"thread_start_us":11556,"threads_started":1}
I20260620 18:46:50.493613 26333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:50.471090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51762 (local address 127.5.225.129:39057)
0620 18:46:50.481856 (+ 10766us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:50.481871 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:50.483037 (+ 1166us) server_negotiation.cc:407] Connection header received
0620 18:46:50.483189 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:50.483201 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:50.483550 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:50.483806 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:46:50.485093 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:50.488285 (+ 3192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:50.491301 (+ 3016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:50.492296 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:50.492449 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:50.492735 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:50.493067 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 18:46:50.493235 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10504,"thread_start_us":10786,"threads_started":1}
I20260620 18:46:50.496062 26335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:50.484106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.130:44333 (local address 127.0.0.1:52998)
0620 18:46:50.487653 (+ 3547us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:50.487670 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:50.487790 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:50.488376 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:50.488382 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:50.488416 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:46:50.488870 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:50.488884 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:50.490674 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:50.490679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:50.492977 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:50.492991 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:50.495154 (+ 2163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:50.495161 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:50.495378 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:50.495473 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 18:46:50.495560 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 18:46:50.495719 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3434,"mutex_wait_us":101,"thread_start_us":3275,"threads_started":1}
I20260620 18:46:50.496225 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:50.483438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52998 (local address 127.5.225.130:44333)
0620 18:46:50.487372 (+ 3934us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:50.487378 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:50.487797 (+ 419us) server_negotiation.cc:407] Connection header received
0620 18:46:50.487911 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:50.487918 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:50.488165 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:50.488384 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:46:50.489072 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:50.490495 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:50.493189 (+ 2694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:50.494654 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:50.494784 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:50.495008 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:50.495685 (+ 677us) server_negotiation.cc:299] Negotiation successful
0620 18:46:50.495848 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3782,"thread_start_us":3661,"threads_started":1}
I20260620 18:46:50.496245 26332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:50.470793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56462 (local address 127.5.225.131:40617)
0620 18:46:50.487969 (+ 17176us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:50.487977 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:50.488005 (+ 28us) server_negotiation.cc:407] Connection header received
0620 18:46:50.488090 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:50.488096 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:50.488312 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:50.488481 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:46:50.489461 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:50.491077 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:50.493552 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:50.494960 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:50.495106 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:50.495375 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:50.495709 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 18:46:50.495877 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16984,"thread_start_us":17053,"threads_started":1}
I20260620 18:46:50.496793 26330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:50.470500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.131:40617 (local address 127.0.0.1:56462)
0620 18:46:50.482475 (+ 11975us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:50.482528 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:50.482726 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:50.488739 (+ 6013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:50.488742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:50.488765 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:46:50.489248 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:50.489261 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:50.491239 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:50.491244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:50.493346 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:50.493359 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:50.495179 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:50.495185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:50.495431 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:50.495496 (+ 65us) client_negotiation.cc:769] Sending connection context
0620 18:46:50.495596 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 18:46:50.496546 (+ 950us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11571,"mutex_wait_us":825,"thread_start_us":12374,"threads_started":1}
I20260620 18:46:50.498246 25992 tablet_service.cc:1467] Processing CreateTablet for tablet 6b0f5c030c36470ea1e9f82024eb49bb (DEFAULT_TABLE table=test-table [id=1a5ca19fd40a4a018463be742e94e988]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:46:50.499868 25992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b0f5c030c36470ea1e9f82024eb49bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:46:50.500169 26123 tablet_service.cc:1467] Processing CreateTablet for tablet 6b0f5c030c36470ea1e9f82024eb49bb (DEFAULT_TABLE table=test-table [id=1a5ca19fd40a4a018463be742e94e988]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:46:50.501250 26123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b0f5c030c36470ea1e9f82024eb49bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:46:50.506812 26253 tablet_service.cc:1467] Processing CreateTablet for tablet 6b0f5c030c36470ea1e9f82024eb49bb (DEFAULT_TABLE table=test-table [id=1a5ca19fd40a4a018463be742e94e988]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:46:50.508179 26253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b0f5c030c36470ea1e9f82024eb49bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:46:50.514936 26336 tablet_bootstrap.cc:492] T 6b0f5c030c36470ea1e9f82024eb49bb P 4a79ba5b7dd146edbdaea8b3e7a7ec7a: Bootstrap starting.
I20260620 18:46:50.518651 26336 tablet_bootstrap.cc:654] T 6b0f5c030c36470ea1e9f82024eb49bb P 4a79ba5b7dd146edbdaea8b3e7a7ec7a: Neither blocks nor log segments found. Creating new log.
I20260620 18:46:50.519903 26336 log.cc:824] T 6b0f5c030c36470ea1e9f82024eb49bb P 4a79ba5b7dd146edbdaea8b3e7a7ec7a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:46:50.523427 26337 tablet_bootstrap.cc:492] T 6b0f5c030c36470ea1e9f82024eb49bb P 920d1a3abe914aa1ae7d22b643af4cd9: Bootstrap starting.
I20260620 18:46:50.523614 26339 tablet_bootstrap.cc:492] T 6b0f5c030c36470ea1e9f82024eb49bb P dbe157a831e24528b98dcc50a321c00e: Bootstrap starting.
I20260620 18:46:50.524708 26336 tablet_bootstrap.cc:492] T 6b0f5c030c36470ea1e9f82024eb49bb P 4a79ba5b7dd146edbdaea8b3e7a7ec7a: No bootstrap required, opened a new log
I20260620 18:46:50.524983 26336 ts_tablet_manager.cc:1397] T 6b0f5c030c36470ea1e9f82024eb49bb P 4a79ba5b7dd146edbdaea8b3e7a7ec7a: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260620 18:46:50.527096 26337 tablet_bootstrap.cc:654] T 6b0f5c030c36470ea1e9f82024eb49bb P 920d1a3abe914aa1ae7d22b643af4cd9: Neither blocks nor log segments found. Creating new log.
I20260620 18:46:50.527088 26339 tablet_bootstrap.cc:654] T 6b0f5c030c36470ea1e9f82024eb49bb P dbe157a831e24528b98dcc50a321c00e: Neither blocks nor log segments found. Creating new log.
I20260620 18:46:50.528261 26057 heartbeater.cc:503] Master 127.5.225.190:42779 was elected leader, sending a full tablet report...
I20260620 18:46:50.528333 26337 log.cc:824] T 6b0f5c030c36470ea1e9f82024eb49bb P 920d1a3abe914aa1ae7d22b643af4cd9: Log is configured to *not* fsync() on all Append() calls
I20260620 18:46:50.528352 26339 log.cc:824] T 6b0f5c030c36470ea1e9f82024eb49bb P dbe157a831e24528b98dcc50a321c00e: Log is configured to *not* fsync() on all Append() calls
I20260620 18:46:50.534313 26336 raft_consensus.cc:348] T 6b0f5c030c36470ea1e9f82024eb49bb P 4a79ba5b7dd146edbdaea8b3e7a7ec7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920d1a3abe914aa1ae7d22b643af4cd9" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 40617 } } peers { permanent_uuid: "dbe157a831e24528b98dcc50a321c00e" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 39057 } } peers { permanent_uuid: "4a79ba5b7dd146edbdaea8b3e7a7ec7a" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 44333 } }
I20260620 18:46:50.534765 26336 raft_consensus.cc:374] T 6b0f5c030c36470ea1e9f82024eb49bb P 4a79ba5b7dd146edbdaea8b3e7a7ec7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:46:50.534894 26336 raft_consensus.cc:729] T 6b0f5c030c36470ea1e9f82024eb49bb P 4a79ba5b7dd146edbdaea8b3e7a7ec7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a79ba5b7dd146edbdaea8b3e7a7ec7a, State: Initialized, Role: FOLLOWER
I20260620 18:46:50.535488 26336 consensus_queue.cc:260] T 6b0f5c030c36470ea1e9f82024eb49bb P 4a79ba5b7dd146edbdaea8b3e7a7ec7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920d1a3abe914aa1ae7d22b643af4cd9" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 40617 } } peers { permanent_uuid: "dbe157a831e24528b98dcc50a321c00e" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 39057 } } peers { permanent_uuid: "4a79ba5b7dd146edbdaea8b3e7a7ec7a" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 44333 } }
I20260620 18:46:50.536258 26337 tablet_bootstrap.cc:492] T 6b0f5c030c36470ea1e9f82024eb49bb P 920d1a3abe914aa1ae7d22b643af4cd9: No bootstrap required, opened a new log
I20260620 18:46:50.536262 26339 tablet_bootstrap.cc:492] T 6b0f5c030c36470ea1e9f82024eb49bb P dbe157a831e24528b98dcc50a321c00e: No bootstrap required, opened a new log
I20260620 18:46:50.536516 26188 heartbeater.cc:503] Master 127.5.225.190:42779 was elected leader, sending a full tablet report...
I20260620 18:46:50.536541 26339 ts_tablet_manager.cc:1397] T 6b0f5c030c36470ea1e9f82024eb49bb P dbe157a831e24528b98dcc50a321c00e: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
I20260620 18:46:50.536537 26337 ts_tablet_manager.cc:1397] T 6b0f5c030c36470ea1e9f82024eb49bb P 920d1a3abe914aa1ae7d22b643af4cd9: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20260620 18:46:50.538335 26336 ts_tablet_manager.cc:1428] T 6b0f5c030c36470ea1e9f82024eb49bb P 4a79ba5b7dd146edbdaea8b3e7a7ec7a: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260620 18:46:50.547405 26339 raft_consensus.cc:348] T 6b0f5c030c36470ea1e9f82024eb49bb P dbe157a831e24528b98dcc50a321c00e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920d1a3abe914aa1ae7d22b643af4cd9" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 40617 } } peers { permanent_uuid: "dbe157a831e24528b98dcc50a321c00e" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 39057 } } peers { permanent_uuid: "4a79ba5b7dd146edbdaea8b3e7a7ec7a" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 44333 } }
I20260620 18:46:50.547406 26337 raft_consensus.cc:348] T 6b0f5c030c36470ea1e9f82024eb49bb P 920d1a3abe914aa1ae7d22b643af4cd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920d1a3abe914aa1ae7d22b643af4cd9" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 40617 } } peers { permanent_uuid: "dbe157a831e24528b98dcc50a321c00e" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 39057 } } peers { permanent_uuid: "4a79ba5b7dd146edbdaea8b3e7a7ec7a" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 44333 } }
I20260620 18:46:50.547861 26337 raft_consensus.cc:374] T 6b0f5c030c36470ea1e9f82024eb49bb P 920d1a3abe914aa1ae7d22b643af4cd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:46:50.547861 26339 raft_consensus.cc:374] T 6b0f5c030c36470ea1e9f82024eb49bb P dbe157a831e24528b98dcc50a321c00e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:46:50.547968 26339 raft_consensus.cc:729] T 6b0f5c030c36470ea1e9f82024eb49bb P dbe157a831e24528b98dcc50a321c00e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbe157a831e24528b98dcc50a321c00e, State: Initialized, Role: FOLLOWER
I20260620 18:46:50.547968 26337 raft_consensus.cc:729] T 6b0f5c030c36470ea1e9f82024eb49bb P 920d1a3abe914aa1ae7d22b643af4cd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 920d1a3abe914aa1ae7d22b643af4cd9, State: Initialized, Role: FOLLOWER
I20260620 18:46:50.548594 26337 consensus_queue.cc:260] T 6b0f5c030c36470ea1e9f82024eb49bb P 920d1a3abe914aa1ae7d22b643af4cd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920d1a3abe914aa1ae7d22b643af4cd9" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 40617 } } peers { permanent_uuid: "dbe157a831e24528b98dcc50a321c00e" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 39057 } } peers { permanent_uuid: "4a79ba5b7dd146edbdaea8b3e7a7ec7a" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 44333 } }
I20260620 18:46:50.548594 26339 consensus_queue.cc:260] T 6b0f5c030c36470ea1e9f82024eb49bb P dbe157a831e24528b98dcc50a321c00e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920d1a3abe914aa1ae7d22b643af4cd9" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 40617 } } peers { permanent_uuid: "dbe157a831e24528b98dcc50a321c00e" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 39057 } } peers { permanent_uuid: "4a79ba5b7dd146edbdaea8b3e7a7ec7a" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 44333 } }
I20260620 18:46:50.549674 26318 heartbeater.cc:503] Master 127.5.225.190:42779 was elected leader, sending a full tablet report...
I20260620 18:46:50.550509 26337 ts_tablet_manager.cc:1428] T 6b0f5c030c36470ea1e9f82024eb49bb P 920d1a3abe914aa1ae7d22b643af4cd9: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260620 18:46:50.550750 26339 ts_tablet_manager.cc:1428] T 6b0f5c030c36470ea1e9f82024eb49bb P dbe157a831e24528b98dcc50a321c00e: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260620 18:46:50.573864 26342 raft_consensus.cc:482] T 6b0f5c030c36470ea1e9f82024eb49bb P 4a79ba5b7dd146edbdaea8b3e7a7ec7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:46:50.574236 26342 raft_consensus.cc:504] T 6b0f5c030c36470ea1e9f82024eb49bb P 4a79ba5b7dd146edbdaea8b3e7a7ec7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920d1a3abe914aa1ae7d22b643af4cd9" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 40617 } } peers { permanent_uuid: "dbe157a831e24528b98dcc50a321c00e" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 39057 } } peers { permanent_uuid: "4a79ba5b7dd146edbdaea8b3e7a7ec7a" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 44333 } }
I20260620 18:46:50.576004 26342 leader_election.cc:288] T 6b0f5c030c36470ea1e9f82024eb49bb P 4a79ba5b7dd146edbdaea8b3e7a7ec7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 920d1a3abe914aa1ae7d22b643af4cd9 (127.5.225.131:40617), dbe157a831e24528b98dcc50a321c00e (127.5.225.129:39057)
I20260620 18:46:50.582793 26333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:50.576517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:52405 (local address 127.5.225.129:39057)
0620 18:46:50.576732 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:50.576740 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:50.577158 (+ 418us) server_negotiation.cc:407] Connection header received
0620 18:46:50.577414 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:50.577418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:50.577488 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:50.577599 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:46:50.578124 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:50.578878 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:50.580796 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:50.581831 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:50.582009 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:50.582079 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:50.582420 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 18:46:50.582541 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 18:46:50.582882 26345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:50.576281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:39057 (local address 127.5.225.130:52405)
0620 18:46:50.577014 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:50.577038 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:50.577259 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:50.577615 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:50.577620 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:50.577646 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:46:50.577975 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:50.577984 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:50.579019 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:50.579024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:50.580585 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:50.580602 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:50.582020 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:50.582026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:50.582258 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:50.582283 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 18:46:50.582394 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 18:46:50.582522 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":2,"thread_start_us":463,"threads_started":1}
I20260620 18:46:50.582917 26332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:50.576280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.225.130:48859 (local address 127.5.225.131:40617)
0620 18:46:50.576500 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:50.576510 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:50.577522 (+ 1012us) server_negotiation.cc:407] Connection header received
0620 18:46:50.577715 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:50.577718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:50.577776 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:50.577872 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:46:50.578470 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:50.579161 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:50.581066 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:50.582001 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:50.582211 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:50.582279 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:50.582567 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 18:46:50.582689 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:46:50.583129 26346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:50.576338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.131:40617 (local address 127.5.225.130:48859)
0620 18:46:50.577423 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:50.577435 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:50.577563 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:50.577888 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:50.577892 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:50.577918 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:46:50.578325 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:50.578334 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:50.579301 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:50.579306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:50.580878 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:50.580892 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:50.582215 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:50.582222 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:50.582413 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:50.582436 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 18:46:50.582540 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 18:46:50.582672 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":964,"thread_start_us":689,"threads_started":1}
I20260620 18:46:50.584105 26273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0f5c030c36470ea1e9f82024eb49bb" candidate_uuid: "4a79ba5b7dd146edbdaea8b3e7a7ec7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "920d1a3abe914aa1ae7d22b643af4cd9" is_pre_election: true
I20260620 18:46:50.584182 26012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0f5c030c36470ea1e9f82024eb49bb" candidate_uuid: "4a79ba5b7dd146edbdaea8b3e7a7ec7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbe157a831e24528b98dcc50a321c00e" is_pre_election: true
I20260620 18:46:50.584626 26273 raft_consensus.cc:2454] T 6b0f5c030c36470ea1e9f82024eb49bb P 920d1a3abe914aa1ae7d22b643af4cd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a79ba5b7dd146edbdaea8b3e7a7ec7a in term 0.
I20260620 18:46:50.584674 26012 raft_consensus.cc:2454] T 6b0f5c030c36470ea1e9f82024eb49bb P dbe157a831e24528b98dcc50a321c00e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a79ba5b7dd146edbdaea8b3e7a7ec7a in term 0.
I20260620 18:46:50.585253 26078 leader_election.cc:302] T 6b0f5c030c36470ea1e9f82024eb49bb P 4a79ba5b7dd146edbdaea8b3e7a7ec7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a79ba5b7dd146edbdaea8b3e7a7ec7a, 920d1a3abe914aa1ae7d22b643af4cd9; no voters:
I20260620 18:46:50.585634 26342 raft_consensus.cc:2781] T 6b0f5c030c36470ea1e9f82024eb49bb P 4a79ba5b7dd146edbdaea8b3e7a7ec7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:46:50.585781 26342 raft_consensus.cc:482] T 6b0f5c030c36470ea1e9f82024eb49bb P 4a79ba5b7dd146edbdaea8b3e7a7ec7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:46:50.585886 26342 raft_consensus.cc:3037] T 6b0f5c030c36470ea1e9f82024eb49bb P 4a79ba5b7dd146edbdaea8b3e7a7ec7a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:46:50.588104 26342 raft_consensus.cc:504] T 6b0f5c030c36470ea1e9f82024eb49bb P 4a79ba5b7dd146edbdaea8b3e7a7ec7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920d1a3abe914aa1ae7d22b643af4cd9" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 40617 } } peers { permanent_uuid: "dbe157a831e24528b98dcc50a321c00e" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 39057 } } peers { permanent_uuid: "4a79ba5b7dd146edbdaea8b3e7a7ec7a" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 44333 } }
I20260620 18:46:50.588758 26342 leader_election.cc:288] T 6b0f5c030c36470ea1e9f82024eb49bb P 4a79ba5b7dd146edbdaea8b3e7a7ec7a [CANDIDATE]: Term 1 election: Requested vote from peers 920d1a3abe914aa1ae7d22b643af4cd9 (127.5.225.131:40617), dbe157a831e24528b98dcc50a321c00e (127.5.225.129:39057)
I20260620 18:46:50.589112 26273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0f5c030c36470ea1e9f82024eb49bb" candidate_uuid: "4a79ba5b7dd146edbdaea8b3e7a7ec7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "920d1a3abe914aa1ae7d22b643af4cd9"
I20260620 18:46:50.589267 26012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0f5c030c36470ea1e9f82024eb49bb" candidate_uuid: "4a79ba5b7dd146edbdaea8b3e7a7ec7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbe157a831e24528b98dcc50a321c00e"
I20260620 18:46:50.589309 26273 raft_consensus.cc:3037] T 6b0f5c030c36470ea1e9f82024eb49bb P 920d1a3abe914aa1ae7d22b643af4cd9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:46:50.589478 26012 raft_consensus.cc:3037] T 6b0f5c030c36470ea1e9f82024eb49bb P dbe157a831e24528b98dcc50a321c00e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:46:50.591583 26273 raft_consensus.cc:2454] T 6b0f5c030c36470ea1e9f82024eb49bb P 920d1a3abe914aa1ae7d22b643af4cd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a79ba5b7dd146edbdaea8b3e7a7ec7a in term 1.
I20260620 18:46:50.591759 26012 raft_consensus.cc:2454] T 6b0f5c030c36470ea1e9f82024eb49bb P dbe157a831e24528b98dcc50a321c00e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a79ba5b7dd146edbdaea8b3e7a7ec7a in term 1.
I20260620 18:46:50.591970 26078 leader_election.cc:302] T 6b0f5c030c36470ea1e9f82024eb49bb P 4a79ba5b7dd146edbdaea8b3e7a7ec7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a79ba5b7dd146edbdaea8b3e7a7ec7a, 920d1a3abe914aa1ae7d22b643af4cd9; no voters:
I20260620 18:46:50.592223 26342 raft_consensus.cc:2781] T 6b0f5c030c36470ea1e9f82024eb49bb P 4a79ba5b7dd146edbdaea8b3e7a7ec7a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:46:50.592612 26342 raft_consensus.cc:686] T 6b0f5c030c36470ea1e9f82024eb49bb P 4a79ba5b7dd146edbdaea8b3e7a7ec7a [term 1 LEADER]: Becoming Leader. State: Replica: 4a79ba5b7dd146edbdaea8b3e7a7ec7a, State: Running, Role: LEADER
I20260620 18:46:50.593106 26342 consensus_queue.cc:237] T 6b0f5c030c36470ea1e9f82024eb49bb P 4a79ba5b7dd146edbdaea8b3e7a7ec7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920d1a3abe914aa1ae7d22b643af4cd9" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 40617 } } peers { permanent_uuid: "dbe157a831e24528b98dcc50a321c00e" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 39057 } } peers { permanent_uuid: "4a79ba5b7dd146edbdaea8b3e7a7ec7a" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 44333 } }
I20260620 18:46:50.597445 25870 catalog_manager.cc:5495] T 6b0f5c030c36470ea1e9f82024eb49bb P 4a79ba5b7dd146edbdaea8b3e7a7ec7a reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a79ba5b7dd146edbdaea8b3e7a7ec7a (127.5.225.130). New cstate: current_term: 1 leader_uuid: "4a79ba5b7dd146edbdaea8b3e7a7ec7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920d1a3abe914aa1ae7d22b643af4cd9" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 40617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbe157a831e24528b98dcc50a321c00e" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 39057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a79ba5b7dd146edbdaea8b3e7a7ec7a" member_type: VOTER last_known_addr { host: "127.5.225.130" port: 44333 } health_report { overall_health: HEALTHY } } }
I20260620 18:46:50.616550 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:50.611638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53006 (local address 127.5.225.130:44333)
0620 18:46:50.611802 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:50.611807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:50.611823 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:46:50.611975 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:50.611978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:50.612051 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:50.612179 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:46:50.612477 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:50.613176 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:50.614533 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:50.614932 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:50.615025 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:50.615444 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:46:50.615932 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:46:50.616046 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:50.616189 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 18:46:50.616248 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 18:46:50.624470 26012 raft_consensus.cc:1261] T 6b0f5c030c36470ea1e9f82024eb49bb P dbe157a831e24528b98dcc50a321c00e [term 1 FOLLOWER]: Refusing update from remote peer 4a79ba5b7dd146edbdaea8b3e7a7ec7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:46:50.624470 26273 raft_consensus.cc:1261] T 6b0f5c030c36470ea1e9f82024eb49bb P 920d1a3abe914aa1ae7d22b643af4cd9 [term 1 FOLLOWER]: Refusing update from remote peer 4a79ba5b7dd146edbdaea8b3e7a7ec7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:46:50.625437 26347 consensus_queue.cc:1035] T 6b0f5c030c36470ea1e9f82024eb49bb P 4a79ba5b7dd146edbdaea8b3e7a7ec7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbe157a831e24528b98dcc50a321c00e" member_type: VOTER last_known_addr { host: "127.5.225.129" port: 39057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:46:50.625748 26342 consensus_queue.cc:1035] T 6b0f5c030c36470ea1e9f82024eb49bb P 4a79ba5b7dd146edbdaea8b3e7a7ec7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "920d1a3abe914aa1ae7d22b643af4cd9" member_type: VOTER last_known_addr { host: "127.5.225.131" port: 40617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:46:50.637602 26351 mvcc.cc:205] Tried to move back new op lower bound from 7298995038707703808 to 7298995038593437696. Current Snapshot: MvccSnapshot[applied={T|T < 7298995038707703808}]
I20260620 18:46:50.648723 25870 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:46:50.649029 25870 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:46:50.652803 25870 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 110a6e6b7db0488f8b448ce28bbcffbd: Sending DeleteTablet for 3 replicas of tablet 6b0f5c030c36470ea1e9f82024eb49bb
I20260620 18:46:50.653923 26253 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b0f5c030c36470ea1e9f82024eb49bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:46:50 UTC) from {username='slave'} at 127.0.0.1:56462
I20260620 18:46:50.654143 25992 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b0f5c030c36470ea1e9f82024eb49bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:46:50 UTC) from {username='slave'} at 127.0.0.1:51762
I20260620 18:46:50.654788 26123 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b0f5c030c36470ea1e9f82024eb49bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:46:50 UTC) from {username='slave'} at 127.0.0.1:52998
I20260620 18:46:50.660089 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 25930
I20260620 18:46:50.665690 26361 tablet_replica.cc:330] T 6b0f5c030c36470ea1e9f82024eb49bb P 4a79ba5b7dd146edbdaea8b3e7a7ec7a: stopping tablet replica
I20260620 18:46:50.666906 26361 raft_consensus.cc:2229] T 6b0f5c030c36470ea1e9f82024eb49bb P 4a79ba5b7dd146edbdaea8b3e7a7ec7a [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:46:50.670948 26361 raft_consensus.cc:2258] T 6b0f5c030c36470ea1e9f82024eb49bb P 4a79ba5b7dd146edbdaea8b3e7a7ec7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:46:50.675889 26361 ts_tablet_manager.cc:1905] T 6b0f5c030c36470ea1e9f82024eb49bb P 4a79ba5b7dd146edbdaea8b3e7a7ec7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:46:50.678117 26360 tablet_replica.cc:330] T 6b0f5c030c36470ea1e9f82024eb49bb P dbe157a831e24528b98dcc50a321c00e: stopping tablet replica
I20260620 18:46:50.679368 26360 raft_consensus.cc:2229] T 6b0f5c030c36470ea1e9f82024eb49bb P dbe157a831e24528b98dcc50a321c00e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:46:50.682541 26333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:50.661201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51774 (local address 127.5.225.129:39057)
0620 18:46:50.661384 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:50.661391 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:50.664262 (+ 2871us) server_negotiation.cc:407] Connection header received
0620 18:46:50.664496 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:50.664503 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:50.664635 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:50.664817 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:50.667201 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:50.668722 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:50.675395 (+ 6673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:50.676357 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:50.676522 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:50.679797 (+ 3275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:50.679814 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:50.679838 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:46:50.680812 (+ 974us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:50.680891 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:46:50.680932 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:50.681081 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:50.681852 (+ 771us) server_negotiation.cc:299] Negotiation successful
0620 18:46:50.682019 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:46:50.683516 26361 ts_tablet_manager.cc:1918] T 6b0f5c030c36470ea1e9f82024eb49bb P 4a79ba5b7dd146edbdaea8b3e7a7ec7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:46:50.684134 26052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:46:50.684130 26359 tablet_replica.cc:330] T 6b0f5c030c36470ea1e9f82024eb49bb P 920d1a3abe914aa1ae7d22b643af4cd9: stopping tablet replica
I20260620 18:46:50.684540 26359 raft_consensus.cc:2229] T 6b0f5c030c36470ea1e9f82024eb49bb P 920d1a3abe914aa1ae7d22b643af4cd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:46:50.684974 26361 log.cc:1196] T 6b0f5c030c36470ea1e9f82024eb49bb P 4a79ba5b7dd146edbdaea8b3e7a7ec7a: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-1/wal/wals/6b0f5c030c36470ea1e9f82024eb49bb
I20260620 18:46:50.685019 26359 raft_consensus.cc:2258] T 6b0f5c030c36470ea1e9f82024eb49bb P 920d1a3abe914aa1ae7d22b643af4cd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:46:50.685696 26361 ts_tablet_manager.cc:1939] T 6b0f5c030c36470ea1e9f82024eb49bb P 4a79ba5b7dd146edbdaea8b3e7a7ec7a: Deleting consensus metadata
I20260620 18:46:50.687253 26359 ts_tablet_manager.cc:1905] T 6b0f5c030c36470ea1e9f82024eb49bb P 920d1a3abe914aa1ae7d22b643af4cd9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:46:50.687985 25855 catalog_manager.cc:4841] TS 4a79ba5b7dd146edbdaea8b3e7a7ec7a (127.5.225.130:44333): tablet 6b0f5c030c36470ea1e9f82024eb49bb (table test-table [id=1a5ca19fd40a4a018463be742e94e988]) successfully deleted
I20260620 18:46:50.694149 26359 ts_tablet_manager.cc:1918] T 6b0f5c030c36470ea1e9f82024eb49bb P 920d1a3abe914aa1ae7d22b643af4cd9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:46:50.694363 26359 log.cc:1196] T 6b0f5c030c36470ea1e9f82024eb49bb P 920d1a3abe914aa1ae7d22b643af4cd9: Deleting WAL directory at /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980894474392-6022-0/minicluster-data/ts-2/wal/wals/6b0f5c030c36470ea1e9f82024eb49bb
I20260620 18:46:50.694866 26359 ts_tablet_manager.cc:1939] T 6b0f5c030c36470ea1e9f82024eb49bb P 920d1a3abe914aa1ae7d22b643af4cd9: Deleting consensus metadata
I20260620 18:46:50.696674 25857 catalog_manager.cc:4841] TS 920d1a3abe914aa1ae7d22b643af4cd9 (127.5.225.131:40617): tablet 6b0f5c030c36470ea1e9f82024eb49bb (table test-table [id=1a5ca19fd40a4a018463be742e94e988]) successfully deleted
I20260620 18:46:50.778369 26360 raft_consensus.cc:2258] T 6b0f5c030c36470ea1e9f82024eb49bb P dbe157a831e24528b98dcc50a321c00e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:46:50.779093 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 25930
W20260620 18:46:50.804754 25855 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.225.129:39057 (error 108)
I20260620 18:46:50.804981 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 26061
I20260620 18:46:50.807016 26330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:50.806512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:39057 (local address 127.0.0.1:51786)
0620 18:46:50.806686 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:50.806828 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.129:39057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 18:46:50.807317 25855 catalog_manager.cc:4584] TS dbe157a831e24528b98dcc50a321c00e (127.5.225.129:39057): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6b0f5c030c36470ea1e9f82024eb49bb: Network error: Client connection negotiation failed: client connection to 127.5.225.129:39057: connect: Connection refused (error 111)
I20260620 18:46:50.813364 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:50.805967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53020 (local address 127.5.225.130:44333)
0620 18:46:50.806225 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:50.806234 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:50.806257 (+ 23us) server_negotiation.cc:407] Connection header received
0620 18:46:50.806383 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:50.806388 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:50.806497 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:50.806614 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:50.807239 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:50.808159 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:50.810118 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:50.810610 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:50.810717 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:50.811865 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:50.811874 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:50.811890 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:46:50.812533 (+ 643us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:50.812581 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:46:50.812600 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:50.812691 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:50.812875 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 18:46:50.812969 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:46:50.814065 26183 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:46:50.861156 26330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:50.860647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:39057 (local address 127.0.0.1:51794)
0620 18:46:50.860864 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:50.860969 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.129:39057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 18:46:50.862416 26330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:50.861951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:39057 (local address 127.0.0.1:51796)
0620 18:46:50.862123 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:50.862264 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.129:39057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 18:46:50.923038 26330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:50.922545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:39057 (local address 127.0.0.1:51812)
0620 18:46:50.922783 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:50.922877 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.129:39057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 18:46:50.924146 26330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:50.923806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:39057 (local address 127.0.0.1:51818)
0620 18:46:50.923940 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:50.924012 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.129:39057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 18:46:50.926432 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 26061
I20260620 18:46:50.944883 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 26191
I20260620 18:46:50.951476 26332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:50.945635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56468 (local address 127.5.225.131:40617)
0620 18:46:50.945838 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:50.945843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:50.945861 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:46:50.945957 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:50.945961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:50.946051 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:50.946136 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:50.946594 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:50.947313 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:50.948503 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:50.948939 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:50.949043 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:50.949987 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:50.949996 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:50.950011 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:46:50.950763 (+ 752us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:50.950810 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:46:50.950830 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:50.950915 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:50.951048 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 18:46:50.951131 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 18:46:50.952162 26313 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:46:51.005103 26330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:51.004551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:39057 (local address 127.0.0.1:51832)
0620 18:46:51.004767 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:51.004883 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.129:39057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 18:46:51.006240 26330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:51.005764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.129:39057 (local address 127.0.0.1:51842)
0620 18:46:51.005985 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:51.006081 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.129:39057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260620 18:46:51.048065 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 26191
I20260620 18:46:51.066715 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 25840
I20260620 18:46:51.067883 25901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:46:51.178831 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 25840
2026-06-20T18:46:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2730 ms)
[----------] 6 tests from EncryptionPolicyTest (21260 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:46:51 dist-test-slave-k6lg krb5kdc[26370](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:46:51 dist-test-slave-k6lg krb5kdc[26370](info): set up 2 sockets
Jun 20 18:46:51 dist-test-slave-k6lg krb5kdc[26370](info): commencing operation
krb5kdc: starting...
W20260620 18:46:52.286758 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.004s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:46:52 dist-test-slave-k6lg krb5kdc[26370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:46:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:46:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:46:52.529592 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41427
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:46:52.862562 26383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:46:52.862847 26383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:46:52.862939 26383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:46:52.870221 26383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:46:52.870335 26383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:46:52.870381 26383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:46:52.870441 26383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:46:52.870484 26383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:46:52.880117 26383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41427
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:46:52.881320 26383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:46:52.882565 26383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:46:52.894377 26388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:52.894411 26389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:52.894340 26391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:46:52.895008 26383 server_base.cc:1034] running on GCE node
I20260620 18:46:52.895778 26383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:46:52.897056 26383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:46:52.898285 26383 hybrid_clock.cc:648] HybridClock initialized: now 1781981212898235 us; error 33 us; skew 500 ppm
I20260620 18:46:52.898584 26383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:46:52 dist-test-slave-k6lg krb5kdc[26370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981212, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:46:52.905040 26383 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:46:52.906474 26383 webserver.cc:460] Webserver started at http://127.0.0.1:34191/ using document root <none> and password file <none>
I20260620 18:46:52.907109 26383 fs_manager.cc:359] Metadata directory not provided
I20260620 18:46:52.907205 26383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:46:52.907441 26383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:46:52.910342 26383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "4843af6e3a414e6fa6aba606bfa38f0e"
format_stamp: "Formatted at 2026-06-20 18:46:52 on dist-test-slave-k6lg"
server_key: "ec9d636747965895cf3eff8eff3d61fb"
server_key_iv: "53e5634b776c6f9672fa053ccd66406a"
server_key_version: "encryptionkey@0"
I20260620 18:46:52.911051 26383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "4843af6e3a414e6fa6aba606bfa38f0e"
format_stamp: "Formatted at 2026-06-20 18:46:52 on dist-test-slave-k6lg"
server_key: "ec9d636747965895cf3eff8eff3d61fb"
server_key_iv: "53e5634b776c6f9672fa053ccd66406a"
server_key_version: "encryptionkey@0"
I20260620 18:46:52.915872 26383 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 18:46:52.919098 26398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:52.920365 26383 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:46:52.920540 26383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "4843af6e3a414e6fa6aba606bfa38f0e"
format_stamp: "Formatted at 2026-06-20 18:46:52 on dist-test-slave-k6lg"
server_key: "ec9d636747965895cf3eff8eff3d61fb"
server_key_iv: "53e5634b776c6f9672fa053ccd66406a"
server_key_version: "encryptionkey@0"
I20260620 18:46:52.920737 26383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:46:52.956943 26383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:46:52.961045 26383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:46:52.961351 26383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:46:52.980257 26383 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41427
I20260620 18:46:52.980304 26449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41427 every 8 connection(s)
I20260620 18:46:52.981668 26383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:46:52.984412 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 26383
I20260620 18:46:52.984642 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:46:52.985445 6022 external_mini_cluster.cc:1428] Setting key c6b7494d6dbc72bfe514d5a4d5174bd1
I20260620 18:46:52.986001 26450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:46:53.000496 26450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4843af6e3a414e6fa6aba606bfa38f0e: Bootstrap starting.
Jun 20 18:46:52 dist-test-slave-k6lg krb5kdc[26370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 18:46:53.004252 26450 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4843af6e3a414e6fa6aba606bfa38f0e: Neither blocks nor log segments found. Creating new log.
I20260620 18:46:53.005659 26450 log.cc:824] T 00000000000000000000000000000000 P 4843af6e3a414e6fa6aba606bfa38f0e: Log is configured to *not* fsync() on all Append() calls
I20260620 18:46:53.009168 26450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4843af6e3a414e6fa6aba606bfa38f0e: No bootstrap required, opened a new log
I20260620 18:46:53.011101 26453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:52.987594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47893 (local address 127.0.0.1:41427)
0620 18:46:52.988390 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:52.988408 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:52.988447 (+ 39us) server_negotiation.cc:407] Connection header received
0620 18:46:52.990683 (+ 2236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:52.990704 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:52.991111 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:52.991450 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:52.993264 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:52.994634 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:52.996452 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:52.997173 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:52.997321 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:53.002224 (+ 4903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:53.002235 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:53.002256 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:46:53.002305 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:53.006289 (+ 3984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:53.007331 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:53.007340 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:53.007348 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:53.007542 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:53.007980 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:53.007986 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:53.007989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:53.008685 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:53.009010 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:53.009381 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 18:46:53.009759 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":368,"threads_started":1}
I20260620 18:46:53.015223 6022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:46:53.018095 26450 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4843af6e3a414e6fa6aba606bfa38f0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4843af6e3a414e6fa6aba606bfa38f0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41427 } }
I20260620 18:46:53.018592 26450 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4843af6e3a414e6fa6aba606bfa38f0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:46:53.018721 26450 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4843af6e3a414e6fa6aba606bfa38f0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4843af6e3a414e6fa6aba606bfa38f0e, State: Initialized, Role: FOLLOWER
I20260620 18:46:53.019349 26450 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4843af6e3a414e6fa6aba606bfa38f0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4843af6e3a414e6fa6aba606bfa38f0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41427 } }
I20260620 18:46:53.019596 26450 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4843af6e3a414e6fa6aba606bfa38f0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:46:53.019733 26450 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4843af6e3a414e6fa6aba606bfa38f0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:46:53.019925 26450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4843af6e3a414e6fa6aba606bfa38f0e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:46:53.023392 26450 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4843af6e3a414e6fa6aba606bfa38f0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4843af6e3a414e6fa6aba606bfa38f0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41427 } }
I20260620 18:46:53.023983 26450 leader_election.cc:302] T 00000000000000000000000000000000 P 4843af6e3a414e6fa6aba606bfa38f0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4843af6e3a414e6fa6aba606bfa38f0e; no voters:
I20260620 18:46:53.024664 26450 leader_election.cc:288] T 00000000000000000000000000000000 P 4843af6e3a414e6fa6aba606bfa38f0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:46:53.024775 26460 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4843af6e3a414e6fa6aba606bfa38f0e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:46:53.026794 26450 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4843af6e3a414e6fa6aba606bfa38f0e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:46:53.030491 26460 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4843af6e3a414e6fa6aba606bfa38f0e [term 1 LEADER]: Becoming Leader. State: Replica: 4843af6e3a414e6fa6aba606bfa38f0e, State: Running, Role: LEADER
I20260620 18:46:53.031374 26460 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4843af6e3a414e6fa6aba606bfa38f0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4843af6e3a414e6fa6aba606bfa38f0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41427 } }
I20260620 18:46:53.039673 26461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4843af6e3a414e6fa6aba606bfa38f0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4843af6e3a414e6fa6aba606bfa38f0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4843af6e3a414e6fa6aba606bfa38f0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41427 } } }
I20260620 18:46:53.040081 26461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4843af6e3a414e6fa6aba606bfa38f0e [sys.catalog]: This master's current role is: LEADER
I20260620 18:46:53.040998 26462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4843af6e3a414e6fa6aba606bfa38f0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4843af6e3a414e6fa6aba606bfa38f0e. Latest consensus state: current_term: 1 leader_uuid: "4843af6e3a414e6fa6aba606bfa38f0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4843af6e3a414e6fa6aba606bfa38f0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41427 } } }
I20260620 18:46:53.041873 26462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4843af6e3a414e6fa6aba606bfa38f0e [sys.catalog]: This master's current role is: LEADER
I20260620 18:46:53.041921 26453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:53.020120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59411 (local address 127.0.0.1:41427)
0620 18:46:53.020335 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:53.020345 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:53.020794 (+ 449us) server_negotiation.cc:407] Connection header received
0620 18:46:53.020956 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:53.020963 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:53.021031 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:53.021119 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:53.023048 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:53.023904 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:53.025867 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:53.026873 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:53.027042 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:53.031028 (+ 3986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:53.031035 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:53.031043 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:46:53.031092 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:53.036184 (+ 5092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:53.037569 (+ 1385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:53.037578 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:53.037584 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:53.037790 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:53.038551 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:53.038560 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:53.038578 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:53.039226 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:53.040877 (+ 1651us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:53.041110 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 18:46:53.041365 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:46:53.044409 26474 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:46:53.048873 26474 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:46:53.058022 26474 catalog_manager.cc:1347] Generated new cluster ID: 078f235c325d443dbba2dcac0be0584e
I20260620 18:46:53.058139 26474 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:46:53.073382 26474 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:46:53.075268 26474 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:46:53.086383 26474 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4843af6e3a414e6fa6aba606bfa38f0e: Generated new TSK 0
I20260620 18:46:53.087487 26474 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:46:53.094214 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 26383
I20260620 18:46:53.096210 26445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:46:53.186633 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 26383
W20260620 18:46:53.204438 26368 connection.cc:535] client connection to 127.0.0.1:41427 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41427): Connection reset by peer (error 104)
2026-06-20T18:46:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2003 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:46:53 dist-test-slave-k6lg krb5kdc[26483](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:46:53 dist-test-slave-k6lg krb5kdc[26483](info): set up 2 sockets
Jun 20 18:46:53 dist-test-slave-k6lg krb5kdc[26483](info): commencing operation
krb5kdc: starting...
W20260620 18:46:54.310541 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:46:54 dist-test-slave-k6lg krb5kdc[26483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:46:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:46:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:46:54.557214 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44533
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:46:54.893169 26496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:46:54.893698 26496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:46:54.893967 26496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:46:54.901252 26496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:46:54.901405 26496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:46:54.901504 26496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:46:54.901616 26496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:46:54.901696 26496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:46:54.911557 26496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44533
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:46:54.912817 26496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:46:54.914088 26496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:46:54.925875 26501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:54.926110 26504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:54.925905 26502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:46:54.926640 26496 server_base.cc:1034] running on GCE node
I20260620 18:46:54.927500 26496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:46:54.928874 26496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:46:54.930109 26496 hybrid_clock.cc:648] HybridClock initialized: now 1781981214930066 us; error 66 us; skew 500 ppm
I20260620 18:46:54.930488 26496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:46:54 dist-test-slave-k6lg krb5kdc[26483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:46:54.933569 26496 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:46:54.935072 26496 webserver.cc:460] Webserver started at http://127.0.0.1:40273/ using document root <none> and password file <none>
I20260620 18:46:54.935781 26496 fs_manager.cc:359] Metadata directory not provided
I20260620 18:46:54.935935 26496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:46:54.936242 26496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:46:54.939359 26496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "902c4eb81d924691a7423df91df0c177"
format_stamp: "Formatted at 2026-06-20 18:46:54 on dist-test-slave-k6lg"
server_key: "2cb80d562678dfc8f53e33ce7fccdfef"
server_key_iv: "3f525e214dcba66559eb25f69ef47b48"
server_key_version: "encryptionkey@0"
I20260620 18:46:54.940214 26496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "902c4eb81d924691a7423df91df0c177"
format_stamp: "Formatted at 2026-06-20 18:46:54 on dist-test-slave-k6lg"
server_key: "2cb80d562678dfc8f53e33ce7fccdfef"
server_key_iv: "3f525e214dcba66559eb25f69ef47b48"
server_key_version: "encryptionkey@0"
I20260620 18:46:54.945539 26496 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:46:54.949074 26511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:54.950479 26496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:46:54.950675 26496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "902c4eb81d924691a7423df91df0c177"
format_stamp: "Formatted at 2026-06-20 18:46:54 on dist-test-slave-k6lg"
server_key: "2cb80d562678dfc8f53e33ce7fccdfef"
server_key_iv: "3f525e214dcba66559eb25f69ef47b48"
server_key_version: "encryptionkey@0"
I20260620 18:46:54.950876 26496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:46:54.971262 26496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:46:54.975348 26496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:46:54.975625 26496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:46:54.993654 26496 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44533
I20260620 18:46:54.993685 26562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44533 every 8 connection(s)
I20260620 18:46:54.994985 26496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:46:54.998682 26563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:46:55.001816 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 26496
I20260620 18:46:55.001986 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:46:55.002565 6022 external_mini_cluster.cc:1428] Setting key 0692277c0c52f5e2df1419e455e6f5c5
I20260620 18:46:55.012802 26563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 902c4eb81d924691a7423df91df0c177: Bootstrap starting.
I20260620 18:46:55.016986 26563 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 902c4eb81d924691a7423df91df0c177: Neither blocks nor log segments found. Creating new log.
I20260620 18:46:55.018452 26563 log.cc:824] T 00000000000000000000000000000000 P 902c4eb81d924691a7423df91df0c177: Log is configured to *not* fsync() on all Append() calls
I20260620 18:46:55.019138 26566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:55.004717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58003 (local address 127.0.0.1:44533)
0620 18:46:55.005727 (+ 1010us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:55.005743 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:55.006124 (+ 381us) server_negotiation.cc:407] Connection header received
0620 18:46:55.008135 (+ 2011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:55.008178 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:55.008774 (+ 596us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:55.009251 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:55.009816 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:55.011883 (+ 2067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:55.013423 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:55.014484 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:55.014714 (+ 230us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:55.016007 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:55.016018 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:55.016039 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:46:55.016770 (+ 731us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:55.016882 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:46:55.016917 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:55.017082 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:55.017478 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 18:46:55.017674 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":447,"threads_started":1}
I20260620 18:46:55.021565 26563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 902c4eb81d924691a7423df91df0c177: No bootstrap required, opened a new log
I20260620 18:46:55.023069 6022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:46:55.032114 26563 raft_consensus.cc:348] T 00000000000000000000000000000000 P 902c4eb81d924691a7423df91df0c177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902c4eb81d924691a7423df91df0c177" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44533 } }
I20260620 18:46:55.032577 26563 raft_consensus.cc:374] T 00000000000000000000000000000000 P 902c4eb81d924691a7423df91df0c177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:46:55.032713 26563 raft_consensus.cc:729] T 00000000000000000000000000000000 P 902c4eb81d924691a7423df91df0c177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 902c4eb81d924691a7423df91df0c177, State: Initialized, Role: FOLLOWER
I20260620 18:46:55.033465 26563 consensus_queue.cc:260] T 00000000000000000000000000000000 P 902c4eb81d924691a7423df91df0c177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902c4eb81d924691a7423df91df0c177" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44533 } }
I20260620 18:46:55.033762 26563 raft_consensus.cc:388] T 00000000000000000000000000000000 P 902c4eb81d924691a7423df91df0c177 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:46:55.033954 26563 raft_consensus.cc:482] T 00000000000000000000000000000000 P 902c4eb81d924691a7423df91df0c177 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:46:55.034143 26563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 902c4eb81d924691a7423df91df0c177 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:46:55.035291 26566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:55.028428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44803 (local address 127.0.0.1:44533)
0620 18:46:55.028646 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:55.028658 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:55.029031 (+ 373us) server_negotiation.cc:407] Connection header received
0620 18:46:55.029187 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:55.029193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:55.029271 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:55.029377 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:55.029884 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:55.031129 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:55.032501 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:55.033181 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:55.033314 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:55.034451 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:55.034456 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:55.034459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:46:55.034477 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:55.034533 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:46:55.034549 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:55.034643 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:55.034867 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 18:46:55.034914 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:46:55.037209 26563 raft_consensus.cc:504] T 00000000000000000000000000000000 P 902c4eb81d924691a7423df91df0c177 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902c4eb81d924691a7423df91df0c177" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44533 } }
I20260620 18:46:55.037727 26563 leader_election.cc:302] T 00000000000000000000000000000000 P 902c4eb81d924691a7423df91df0c177 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 902c4eb81d924691a7423df91df0c177; no voters:
I20260620 18:46:55.038338 26563 leader_election.cc:288] T 00000000000000000000000000000000 P 902c4eb81d924691a7423df91df0c177 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:46:55.038461 26573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 902c4eb81d924691a7423df91df0c177 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:46:55.039219 26573 raft_consensus.cc:686] T 00000000000000000000000000000000 P 902c4eb81d924691a7423df91df0c177 [term 1 LEADER]: Becoming Leader. State: Replica: 902c4eb81d924691a7423df91df0c177, State: Running, Role: LEADER
I20260620 18:46:55.039743 26573 consensus_queue.cc:237] T 00000000000000000000000000000000 P 902c4eb81d924691a7423df91df0c177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902c4eb81d924691a7423df91df0c177" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44533 } }
I20260620 18:46:55.040472 26563 sys_catalog.cc:564] T 00000000000000000000000000000000 P 902c4eb81d924691a7423df91df0c177 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:46:55.042385 26574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 902c4eb81d924691a7423df91df0c177 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "902c4eb81d924691a7423df91df0c177" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902c4eb81d924691a7423df91df0c177" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44533 } } }
I20260620 18:46:55.042761 26574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 902c4eb81d924691a7423df91df0c177 [sys.catalog]: This master's current role is: LEADER
I20260620 18:46:55.043321 26575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 902c4eb81d924691a7423df91df0c177 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 902c4eb81d924691a7423df91df0c177. Latest consensus state: current_term: 1 leader_uuid: "902c4eb81d924691a7423df91df0c177" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902c4eb81d924691a7423df91df0c177" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44533 } } }
I20260620 18:46:55.043637 26578 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:46:55.043664 26575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 902c4eb81d924691a7423df91df0c177 [sys.catalog]: This master's current role is: LEADER
I20260620 18:46:55.049952 26578 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:46:55.058832 26578 catalog_manager.cc:1347] Generated new cluster ID: 8f4075093d3f4399b549da19e67a867c
I20260620 18:46:55.058959 26578 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:46:55.073388 26578 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:46:55.074949 26578 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:46:55.087674 26578 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 902c4eb81d924691a7423df91df0c177: Generated new TSK 0
I20260620 18:46:55.088459 26578 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:46:55.099027 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 26496
W20260620 18:46:55.100083 26558 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58003
W20260620 18:46:55.100819 6022 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 26496: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:46:55.101016 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 26496
W20260620 18:46:55.118216 26481 connection.cc:535] client connection to 127.0.0.1:44533 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44533): Connection reset by peer (error 104)
2026-06-20T18:46:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1912 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:46:55 dist-test-slave-k6lg krb5kdc[26594](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:46:55 dist-test-slave-k6lg krb5kdc[26594](info): set up 2 sockets
Jun 20 18:46:55 dist-test-slave-k6lg krb5kdc[26594](info): commencing operation
krb5kdc: starting...
W20260620 18:46:56.206638 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.005s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:46:56 dist-test-slave-k6lg krb5kdc[26594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:46:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:46:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:46:56.456435 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44241
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:46:56.788841 26607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:46:56.789104 26607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:46:56.789291 26607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:46:56.796504 26607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:46:56.796633 26607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:46:56.796739 26607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:46:56.796849 26607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:46:56.796909 26607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:46:56.806571 26607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44241
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:46:56.807905 26607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:46:56.809311 26607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:46:56.821188 26615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:56.821270 26613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:56.821223 26612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:46:56.822062 26607 server_base.cc:1034] running on GCE node
I20260620 18:46:56.822836 26607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:46:56.824230 26607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:46:56.825467 26607 hybrid_clock.cc:648] HybridClock initialized: now 1781981216825432 us; error 49 us; skew 500 ppm
I20260620 18:46:56.825824 26607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:46:56 dist-test-slave-k6lg krb5kdc[26594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:46:56.828950 26607 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:46:56.830454 26607 webserver.cc:460] Webserver started at http://127.0.0.1:35031/ using document root <none> and password file <none>
I20260620 18:46:56.831199 26607 fs_manager.cc:359] Metadata directory not provided
I20260620 18:46:56.831353 26607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:46:56.831673 26607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:46:56.834820 26607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "c8b23675918e4e9b9c122c119d1fffd2"
format_stamp: "Formatted at 2026-06-20 18:46:56 on dist-test-slave-k6lg"
server_key: "29ae7472363b333c7b348af8f7c5c67e"
server_key_iv: "013ca888cafbdc1eb0633a5d403c00f2"
server_key_version: "encryptionkey@0"
I20260620 18:46:56.835599 26607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "c8b23675918e4e9b9c122c119d1fffd2"
format_stamp: "Formatted at 2026-06-20 18:46:56 on dist-test-slave-k6lg"
server_key: "29ae7472363b333c7b348af8f7c5c67e"
server_key_iv: "013ca888cafbdc1eb0633a5d403c00f2"
server_key_version: "encryptionkey@0"
I20260620 18:46:56.840755 26607 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:46:56.844187 26622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:56.845450 26607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:46:56.845638 26607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "c8b23675918e4e9b9c122c119d1fffd2"
format_stamp: "Formatted at 2026-06-20 18:46:56 on dist-test-slave-k6lg"
server_key: "29ae7472363b333c7b348af8f7c5c67e"
server_key_iv: "013ca888cafbdc1eb0633a5d403c00f2"
server_key_version: "encryptionkey@0"
I20260620 18:46:56.845853 26607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:46:56.851761 26607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:46:56.855755 26607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:46:56.856052 26607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:46:56.874818 26607 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44241
I20260620 18:46:56.874863 26673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44241 every 8 connection(s)
I20260620 18:46:56.876235 26607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:46:56.879819 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 26607
I20260620 18:46:56.880000 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:46:56.880296 26674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:46:56.880581 6022 external_mini_cluster.cc:1428] Setting key 03845e581c111916511ea0d2ddefec54
I20260620 18:46:56.888309 26677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:56.882558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39143 (local address 127.0.0.1:44241)
0620 18:46:56.883411 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:56.883423 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:56.883467 (+ 44us) server_negotiation.cc:407] Connection header received
0620 18:46:56.884980 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:56.885002 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:56.885338 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:56.885665 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:56.886757 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:56.886764 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:56.886777 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:46:56.886794 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:56.886865 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:46:56.886897 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:56.887064 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:56.887188 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 18:46:56.887383 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":393,"threads_started":1}
I20260620 18:46:56.891903 6022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:46:56.893528 26674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8b23675918e4e9b9c122c119d1fffd2: Bootstrap starting.
I20260620 18:46:56.896688 26674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8b23675918e4e9b9c122c119d1fffd2: Neither blocks nor log segments found. Creating new log.
I20260620 18:46:56.897876 26674 log.cc:824] T 00000000000000000000000000000000 P c8b23675918e4e9b9c122c119d1fffd2: Log is configured to *not* fsync() on all Append() calls
I20260620 18:46:56.898882 26677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:56.896218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58859 (local address 127.0.0.1:44241)
0620 18:46:56.896398 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:56.896404 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:56.896833 (+ 429us) server_negotiation.cc:407] Connection header received
0620 18:46:56.897014 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:56.897021 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:56.897087 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:56.897219 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:56.898080 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:56.898086 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:56.898091 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:46:56.898107 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:56.898205 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:46:56.898226 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:56.898353 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:56.898448 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 18:46:56.898504 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:46:56.900761 26674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8b23675918e4e9b9c122c119d1fffd2: No bootstrap required, opened a new log
I20260620 18:46:56.907925 26674 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8b23675918e4e9b9c122c119d1fffd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8b23675918e4e9b9c122c119d1fffd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44241 } }
I20260620 18:46:56.908318 26674 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8b23675918e4e9b9c122c119d1fffd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:46:56.908424 26674 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8b23675918e4e9b9c122c119d1fffd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8b23675918e4e9b9c122c119d1fffd2, State: Initialized, Role: FOLLOWER
I20260620 18:46:56.909088 26674 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8b23675918e4e9b9c122c119d1fffd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8b23675918e4e9b9c122c119d1fffd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44241 } }
I20260620 18:46:56.909368 26674 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8b23675918e4e9b9c122c119d1fffd2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:46:56.909521 26674 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8b23675918e4e9b9c122c119d1fffd2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:46:56.909694 26674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8b23675918e4e9b9c122c119d1fffd2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:46:56.911847 26674 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8b23675918e4e9b9c122c119d1fffd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8b23675918e4e9b9c122c119d1fffd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44241 } }
I20260620 18:46:56.912322 26674 leader_election.cc:302] T 00000000000000000000000000000000 P c8b23675918e4e9b9c122c119d1fffd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8b23675918e4e9b9c122c119d1fffd2; no voters:
I20260620 18:46:56.912869 26674 leader_election.cc:288] T 00000000000000000000000000000000 P c8b23675918e4e9b9c122c119d1fffd2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:46:56.912998 26684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8b23675918e4e9b9c122c119d1fffd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:46:56.913713 26684 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8b23675918e4e9b9c122c119d1fffd2 [term 1 LEADER]: Becoming Leader. State: Replica: c8b23675918e4e9b9c122c119d1fffd2, State: Running, Role: LEADER
I20260620 18:46:56.914294 26684 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8b23675918e4e9b9c122c119d1fffd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8b23675918e4e9b9c122c119d1fffd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44241 } }
I20260620 18:46:56.915076 26674 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8b23675918e4e9b9c122c119d1fffd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:46:56.917472 26685 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8b23675918e4e9b9c122c119d1fffd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8b23675918e4e9b9c122c119d1fffd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8b23675918e4e9b9c122c119d1fffd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44241 } } }
I20260620 18:46:56.917939 26685 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8b23675918e4e9b9c122c119d1fffd2 [sys.catalog]: This master's current role is: LEADER
I20260620 18:46:56.918418 26686 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8b23675918e4e9b9c122c119d1fffd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8b23675918e4e9b9c122c119d1fffd2. Latest consensus state: current_term: 1 leader_uuid: "c8b23675918e4e9b9c122c119d1fffd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8b23675918e4e9b9c122c119d1fffd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44241 } } }
I20260620 18:46:56.918844 26686 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8b23675918e4e9b9c122c119d1fffd2 [sys.catalog]: This master's current role is: LEADER
I20260620 18:46:56.922636 26692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:46:56.927657 26692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:46:56.936323 26692 catalog_manager.cc:1347] Generated new cluster ID: 2bb8393177754511bce090c6d1df3320
I20260620 18:46:56.936441 26692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:46:56.954339 26692 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:46:56.954768 26692 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:46:56.964550 26692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8b23675918e4e9b9c122c119d1fffd2: Generated new TSK 0
I20260620 18:46:56.965404 26692 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:46:56.976218 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 26607
W20260620 18:46:56.977401 26669 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39143
W20260620 18:46:56.978137 6022 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 26607: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:46:56.978312 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 26607
2026-06-20T18:46:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1877 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:46:57 dist-test-slave-k6lg krb5kdc[26705](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:46:57 dist-test-slave-k6lg krb5kdc[26705](info): set up 2 sockets
Jun 20 18:46:57 dist-test-slave-k6lg krb5kdc[26705](info): commencing operation
krb5kdc: starting...
W20260620 18:46:58.078996 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:46:58 dist-test-slave-k6lg krb5kdc[26705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:46:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:46:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:46:58.324957 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44065
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:46:58.645852 26718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:46:58.646220 26718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:46:58.646400 26718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:46:58.653707 26718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:46:58.653877 26718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:46:58.653996 26718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:46:58.654115 26718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:46:58.654222 26718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:46:58.664269 26718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:46:58.665627 26718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:46:58.666944 26718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:46:58.678488 26723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:58.678717 26726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:58.678767 26724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:46:58.679769 26718 server_base.cc:1034] running on GCE node
I20260620 18:46:58.680490 26718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:46:58.681815 26718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:46:58.683037 26718 hybrid_clock.cc:648] HybridClock initialized: now 1781981218682990 us; error 56 us; skew 500 ppm
I20260620 18:46:58.683408 26718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:46:58 dist-test-slave-k6lg krb5kdc[26705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:46:58.690861 26718 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:46:58.692298 26718 webserver.cc:460] Webserver started at http://127.0.0.1:39449/ using document root <none> and password file <none>
I20260620 18:46:58.692988 26718 fs_manager.cc:359] Metadata directory not provided
I20260620 18:46:58.693140 26718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:46:58.693449 26718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:46:58.696449 26718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "3bce298f721740a1b19089e67e2deac2"
format_stamp: "Formatted at 2026-06-20 18:46:58 on dist-test-slave-k6lg"
server_key: "e77d9c05c1e1a4d7aa89fe7faa837cc1"
server_key_iv: "20715390c87083089453e71587697c08"
server_key_version: "encryptionkey@0"
I20260620 18:46:58.697228 26718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "3bce298f721740a1b19089e67e2deac2"
format_stamp: "Formatted at 2026-06-20 18:46:58 on dist-test-slave-k6lg"
server_key: "e77d9c05c1e1a4d7aa89fe7faa837cc1"
server_key_iv: "20715390c87083089453e71587697c08"
server_key_version: "encryptionkey@0"
I20260620 18:46:58.702289 26718 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260620 18:46:58.705519 26733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:58.706890 26718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 18:46:58.707075 26718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "3bce298f721740a1b19089e67e2deac2"
format_stamp: "Formatted at 2026-06-20 18:46:58 on dist-test-slave-k6lg"
server_key: "e77d9c05c1e1a4d7aa89fe7faa837cc1"
server_key_iv: "20715390c87083089453e71587697c08"
server_key_version: "encryptionkey@0"
I20260620 18:46:58.707302 26718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:46:58.723927 26718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:46:58.728515 26718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:46:58.728813 26718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:46:58.746851 26784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44065 every 8 connection(s)
I20260620 18:46:58.746847 26718 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44065
I20260620 18:46:58.748297 26718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:46:58.751965 26785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:46:58.757436 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 26718
I20260620 18:46:58.757658 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:46:58.758083 6022 external_mini_cluster.cc:1428] Setting key cd57b62febcb8efd80a3d45580a956eb
I20260620 18:46:58.762606 26785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bce298f721740a1b19089e67e2deac2: Bootstrap starting.
I20260620 18:46:58.766420 26785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bce298f721740a1b19089e67e2deac2: Neither blocks nor log segments found. Creating new log.
I20260620 18:46:58.767629 26785 log.cc:824] T 00000000000000000000000000000000 P 3bce298f721740a1b19089e67e2deac2: Log is configured to *not* fsync() on all Append() calls
I20260620 18:46:58.770655 26785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bce298f721740a1b19089e67e2deac2: No bootstrap required, opened a new log
Jun 20 18:46:58 dist-test-slave-k6lg krb5kdc[26705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 18:46:58.779199 26785 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bce298f721740a1b19089e67e2deac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bce298f721740a1b19089e67e2deac2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44065 } }
I20260620 18:46:58.779699 26785 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bce298f721740a1b19089e67e2deac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:46:58.779819 26785 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bce298f721740a1b19089e67e2deac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bce298f721740a1b19089e67e2deac2, State: Initialized, Role: FOLLOWER
I20260620 18:46:58.780507 26785 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bce298f721740a1b19089e67e2deac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bce298f721740a1b19089e67e2deac2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44065 } }
I20260620 18:46:58.780754 26785 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bce298f721740a1b19089e67e2deac2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:46:58.780926 26785 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bce298f721740a1b19089e67e2deac2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:46:58.781114 26785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bce298f721740a1b19089e67e2deac2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:46:58.782059 26788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:58.760001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35941 (local address 127.0.0.1:44065)
0620 18:46:58.760887 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:58.760899 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:58.760942 (+ 43us) server_negotiation.cc:407] Connection header received
0620 18:46:58.762800 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:58.762824 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:58.763234 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:58.763563 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:58.765431 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:58.766872 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:58.768634 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:58.769301 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:58.769493 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:58.773850 (+ 4357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:58.773861 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:58.773878 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:46:58.773924 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:58.777226 (+ 3302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:58.777940 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:58.777948 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:58.777955 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:58.778124 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:58.778613 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:58.778617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:58.778619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:58.779174 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:58.779389 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:58.780023 (+ 634us) server_negotiation.cc:299] Negotiation successful
0620 18:46:58.780621 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":463,"threads_started":1}
I20260620 18:46:58.783993 26785 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bce298f721740a1b19089e67e2deac2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bce298f721740a1b19089e67e2deac2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44065 } }
I20260620 18:46:58.784588 26785 leader_election.cc:302] T 00000000000000000000000000000000 P 3bce298f721740a1b19089e67e2deac2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bce298f721740a1b19089e67e2deac2; no voters:
I20260620 18:46:58.785271 26785 leader_election.cc:288] T 00000000000000000000000000000000 P 3bce298f721740a1b19089e67e2deac2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:46:58.785413 26790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bce298f721740a1b19089e67e2deac2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:46:58.786387 6022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:46:58.786823 26790 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bce298f721740a1b19089e67e2deac2 [term 1 LEADER]: Becoming Leader. State: Replica: 3bce298f721740a1b19089e67e2deac2, State: Running, Role: LEADER
I20260620 18:46:58.787312 26790 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bce298f721740a1b19089e67e2deac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bce298f721740a1b19089e67e2deac2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44065 } }
I20260620 18:46:58.787881 26785 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bce298f721740a1b19089e67e2deac2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:46:58.790326 26792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bce298f721740a1b19089e67e2deac2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bce298f721740a1b19089e67e2deac2. Latest consensus state: current_term: 1 leader_uuid: "3bce298f721740a1b19089e67e2deac2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bce298f721740a1b19089e67e2deac2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44065 } } }
I20260620 18:46:58.790774 26792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bce298f721740a1b19089e67e2deac2 [sys.catalog]: This master's current role is: LEADER
I20260620 18:46:58.791357 26791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bce298f721740a1b19089e67e2deac2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bce298f721740a1b19089e67e2deac2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bce298f721740a1b19089e67e2deac2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44065 } } }
I20260620 18:46:58.791704 26791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bce298f721740a1b19089e67e2deac2 [sys.catalog]: This master's current role is: LEADER
I20260620 18:46:58.801863 26796 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:46:58.808521 26796 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:46:58.831378 26796 catalog_manager.cc:1347] Generated new cluster ID: d44e80950e7e4d83971644a9d53badd9
I20260620 18:46:58.831498 26796 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:46:58.836402 26788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:58.810312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54279 (local address 127.0.0.1:44065)
0620 18:46:58.810522 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:58.810528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:58.817441 (+ 6913us) server_negotiation.cc:407] Connection header received
0620 18:46:58.817720 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:58.817726 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:58.817811 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:58.817894 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:58.820701 (+ 2807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:58.822183 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:58.824662 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:58.825587 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:58.825708 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:58.828707 (+ 2999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:58.828715 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:58.828720 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:46:58.828760 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:58.832667 (+ 3907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:58.833580 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:58.833589 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:58.833593 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:58.833728 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:58.834134 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:58.834138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:58.834141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:58.834578 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:58.835665 (+ 1087us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:58.835821 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 18:46:58.836035 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":82}
I20260620 18:46:58.845131 26796 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:46:58.846539 26796 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:46:58.859373 26796 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bce298f721740a1b19089e67e2deac2: Generated new TSK 0
I20260620 18:46:58.860356 26796 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:46:58.866137 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 26718
I20260620 18:46:58.867533 26780 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:46:58.957033 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 26718
2026-06-20T18:46:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1975 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:46:59 dist-test-slave-k6lg krb5kdc[26817](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:46:59 dist-test-slave-k6lg krb5kdc[26817](info): set up 2 sockets
Jun 20 18:46:59 dist-test-slave-k6lg krb5kdc[26817](info): commencing operation
krb5kdc: starting...
W20260620 18:47:00.066710 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:47:00 dist-test-slave-k6lg krb5kdc[26817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:47:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:47:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:47:00.312994 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40489
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:47:00.643136 26830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:00.643427 26830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:00.643527 26830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:00.650940 26830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:47:00.651062 26830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:00.651110 26830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:00.651167 26830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:47:00.651212 26830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:47:00.661257 26830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:47:00.662533 26830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:00.663801 26830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:00.675851 26835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:00.675889 26838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:00.675930 26836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:00.676915 26830 server_base.cc:1034] running on GCE node
I20260620 18:47:00.677793 26830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:00.679296 26830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:00.680521 26830 hybrid_clock.cc:648] HybridClock initialized: now 1781981220680465 us; error 42 us; skew 500 ppm
I20260620 18:47:00.680925 26830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:00 dist-test-slave-k6lg krb5kdc[26817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:00.684067 26830 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:47:00.685495 26830 webserver.cc:460] Webserver started at http://127.0.0.1:33921/ using document root <none> and password file <none>
I20260620 18:47:00.686247 26830 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:00.686385 26830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:00.686723 26830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:47:00.689738 26830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "f47e5380c25f4c1da7e430ee863cca94"
format_stamp: "Formatted at 2026-06-20 18:47:00 on dist-test-slave-k6lg"
server_key: "1b4430a08e1379c18b84a5521e61d0ce"
server_key_iv: "b2710fc44e2ce644c4b1eb5099998871"
server_key_version: "encryptionkey@0"
I20260620 18:47:00.691069 26830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "f47e5380c25f4c1da7e430ee863cca94"
format_stamp: "Formatted at 2026-06-20 18:47:00 on dist-test-slave-k6lg"
server_key: "1b4430a08e1379c18b84a5521e61d0ce"
server_key_iv: "b2710fc44e2ce644c4b1eb5099998871"
server_key_version: "encryptionkey@0"
I20260620 18:47:00.696445 26830 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260620 18:47:00.699852 26845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:00.701152 26830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:47:00.701328 26830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "f47e5380c25f4c1da7e430ee863cca94"
format_stamp: "Formatted at 2026-06-20 18:47:00 on dist-test-slave-k6lg"
server_key: "1b4430a08e1379c18b84a5521e61d0ce"
server_key_iv: "b2710fc44e2ce644c4b1eb5099998871"
server_key_version: "encryptionkey@0"
I20260620 18:47:00.701514 26830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:00.728298 26830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:00.732240 26830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:00.732532 26830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:00.750733 26830 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40489
I20260620 18:47:00.750771 26896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40489 every 8 connection(s)
I20260620 18:47:00.752076 26830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:47:00.755721 26897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:47:00.757915 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 26830
I20260620 18:47:00.758095 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:47:00.758553 6022 external_mini_cluster.cc:1428] Setting key 316e1a8aa43953eba1ae8f78344bfae4
I20260620 18:47:00.767020 26897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f47e5380c25f4c1da7e430ee863cca94: Bootstrap starting.
I20260620 18:47:00.770120 26897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f47e5380c25f4c1da7e430ee863cca94: Neither blocks nor log segments found. Creating new log.
I20260620 18:47:00.771220 26897 log.cc:824] T 00000000000000000000000000000000 P f47e5380c25f4c1da7e430ee863cca94: Log is configured to *not* fsync() on all Append() calls
I20260620 18:47:00.771484 26900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:00.760359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50351 (local address 127.0.0.1:40489)
0620 18:47:00.761235 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:00.761251 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:00.761294 (+ 43us) server_negotiation.cc:407] Connection header received
0620 18:47:00.763063 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:00.763085 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:00.763480 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:00.763857 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:00.764407 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:00.765705 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:00.767137 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:00.767811 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:00.768024 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:00.769190 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:00.769198 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:00.769211 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:47:00.769709 (+ 498us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:00.769780 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:47:00.769804 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:00.769942 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:00.770197 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 18:47:00.770478 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":409,"threads_started":1}
I20260620 18:47:00.774197 26897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f47e5380c25f4c1da7e430ee863cca94: No bootstrap required, opened a new log
I20260620 18:47:00.775039 6022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:47:00.783018 26897 raft_consensus.cc:348] T 00000000000000000000000000000000 P f47e5380c25f4c1da7e430ee863cca94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f47e5380c25f4c1da7e430ee863cca94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40489 } }
I20260620 18:47:00.783509 26897 raft_consensus.cc:374] T 00000000000000000000000000000000 P f47e5380c25f4c1da7e430ee863cca94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:47:00.783624 26897 raft_consensus.cc:729] T 00000000000000000000000000000000 P f47e5380c25f4c1da7e430ee863cca94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f47e5380c25f4c1da7e430ee863cca94, State: Initialized, Role: FOLLOWER
I20260620 18:47:00.784278 26897 consensus_queue.cc:260] T 00000000000000000000000000000000 P f47e5380c25f4c1da7e430ee863cca94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f47e5380c25f4c1da7e430ee863cca94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40489 } }
I20260620 18:47:00.784581 26897 raft_consensus.cc:388] T 00000000000000000000000000000000 P f47e5380c25f4c1da7e430ee863cca94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:47:00.784767 26897 raft_consensus.cc:482] T 00000000000000000000000000000000 P f47e5380c25f4c1da7e430ee863cca94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:47:00.784947 26897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f47e5380c25f4c1da7e430ee863cca94 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:47:00.786012 26900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:00.779322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48437 (local address 127.0.0.1:40489)
0620 18:47:00.779521 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:00.779526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:00.779815 (+ 289us) server_negotiation.cc:407] Connection header received
0620 18:47:00.779984 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:00.779988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:00.780051 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:00.780169 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:00.780762 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:00.781478 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:00.783340 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:00.783935 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:00.784077 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:00.785327 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:00.785332 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:00.785335 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:47:00.785351 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:00.785385 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:47:00.785403 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:00.785511 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:00.785626 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 18:47:00.785721 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 18:47:00.787824 26897 raft_consensus.cc:504] T 00000000000000000000000000000000 P f47e5380c25f4c1da7e430ee863cca94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f47e5380c25f4c1da7e430ee863cca94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40489 } }
I20260620 18:47:00.788309 26897 leader_election.cc:302] T 00000000000000000000000000000000 P f47e5380c25f4c1da7e430ee863cca94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f47e5380c25f4c1da7e430ee863cca94; no voters:
I20260620 18:47:00.788843 26897 leader_election.cc:288] T 00000000000000000000000000000000 P f47e5380c25f4c1da7e430ee863cca94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:47:00.788968 26907 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f47e5380c25f4c1da7e430ee863cca94 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:47:00.789659 26907 raft_consensus.cc:686] T 00000000000000000000000000000000 P f47e5380c25f4c1da7e430ee863cca94 [term 1 LEADER]: Becoming Leader. State: Replica: f47e5380c25f4c1da7e430ee863cca94, State: Running, Role: LEADER
I20260620 18:47:00.790346 26907 consensus_queue.cc:237] T 00000000000000000000000000000000 P f47e5380c25f4c1da7e430ee863cca94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f47e5380c25f4c1da7e430ee863cca94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40489 } }
I20260620 18:47:00.791078 26897 sys_catalog.cc:564] T 00000000000000000000000000000000 P f47e5380c25f4c1da7e430ee863cca94 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:47:00.793035 26908 sys_catalog.cc:455] T 00000000000000000000000000000000 P f47e5380c25f4c1da7e430ee863cca94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f47e5380c25f4c1da7e430ee863cca94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f47e5380c25f4c1da7e430ee863cca94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40489 } } }
I20260620 18:47:00.793365 26908 sys_catalog.cc:458] T 00000000000000000000000000000000 P f47e5380c25f4c1da7e430ee863cca94 [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:00.795030 26912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:47:00.799638 26912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:47:00.802309 26909 sys_catalog.cc:455] T 00000000000000000000000000000000 P f47e5380c25f4c1da7e430ee863cca94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f47e5380c25f4c1da7e430ee863cca94. Latest consensus state: current_term: 1 leader_uuid: "f47e5380c25f4c1da7e430ee863cca94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f47e5380c25f4c1da7e430ee863cca94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40489 } } }
I20260620 18:47:00.805138 26909 sys_catalog.cc:458] T 00000000000000000000000000000000 P f47e5380c25f4c1da7e430ee863cca94 [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:00.814465 26912 catalog_manager.cc:1347] Generated new cluster ID: d6b8e0bbb78049bfbbb98e4d7fe21384
I20260620 18:47:00.814584 26912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:47:00.862123 26912 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:47:00.864023 26912 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:47:00.872906 26912 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f47e5380c25f4c1da7e430ee863cca94: Generated new TSK 0
I20260620 18:47:00.873842 26912 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:47:00.893330 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 26830
W20260620 18:47:00.894585 26892 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50351
W20260620 18:47:00.895283 6022 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 26830: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:47:00.895429 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 26830
2026-06-20T18:47:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1940 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:47:00 dist-test-slave-k6lg krb5kdc[26928](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:47:00 dist-test-slave-k6lg krb5kdc[26928](info): set up 2 sockets
Jun 20 18:47:00 dist-test-slave-k6lg krb5kdc[26928](info): commencing operation
krb5kdc: starting...
W20260620 18:47:02.006610 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:47:02 dist-test-slave-k6lg krb5kdc[26928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:47:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:47:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:47:02.261893 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37639
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:47:02.595903 26941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:02.596223 26941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:02.596372 26941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:02.603998 26941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:47:02.604118 26941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:02.604223 26941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:02.604323 26941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:47:02.604399 26941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:47:02.614852 26941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:47:02.616211 26941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:02.617565 26941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:02.629182 26947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:02.629230 26949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:02.629175 26946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:02.630035 26941 server_base.cc:1034] running on GCE node
I20260620 18:47:02.630889 26941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:02.632304 26941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:02.633555 26941 hybrid_clock.cc:648] HybridClock initialized: now 1781981222633510 us; error 36 us; skew 500 ppm
I20260620 18:47:02.633913 26941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:02 dist-test-slave-k6lg krb5kdc[26928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:02.637292 26941 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:47:02.638881 26941 webserver.cc:460] Webserver started at http://127.0.0.1:44891/ using document root <none> and password file <none>
I20260620 18:47:02.639606 26941 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:02.639747 26941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:02.640064 26941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:47:02.643124 26941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "f9a714c9104d45a7bdf688d7f3db0673"
format_stamp: "Formatted at 2026-06-20 18:47:02 on dist-test-slave-k6lg"
server_key: "05bcdfcff6f34a25ee02c695b5dbb468"
server_key_iv: "e4af6581ee062ed427d2312065236708"
server_key_version: "encryptionkey@0"
I20260620 18:47:02.643882 26941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "f9a714c9104d45a7bdf688d7f3db0673"
format_stamp: "Formatted at 2026-06-20 18:47:02 on dist-test-slave-k6lg"
server_key: "05bcdfcff6f34a25ee02c695b5dbb468"
server_key_iv: "e4af6581ee062ed427d2312065236708"
server_key_version: "encryptionkey@0"
I20260620 18:47:02.649178 26941 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 18:47:02.652896 26956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:02.654351 26941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:47:02.654532 26941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "f9a714c9104d45a7bdf688d7f3db0673"
format_stamp: "Formatted at 2026-06-20 18:47:02 on dist-test-slave-k6lg"
server_key: "05bcdfcff6f34a25ee02c695b5dbb468"
server_key_iv: "e4af6581ee062ed427d2312065236708"
server_key_version: "encryptionkey@0"
I20260620 18:47:02.654750 26941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:02.660490 26941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:02.664815 26941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:02.665174 26941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:02.683869 26941 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37639
I20260620 18:47:02.683909 27007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37639 every 8 connection(s)
I20260620 18:47:02.685344 26941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:47:02.686514 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 26941
I20260620 18:47:02.686712 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:47:02.687237 6022 external_mini_cluster.cc:1428] Setting key 2f96f5e5dcd9600fc428ecbf9ff19e42
I20260620 18:47:02.690124 27008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:47:02.695206 27011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:02.689126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53977 (local address 127.0.0.1:37639)
0620 18:47:02.689897 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:02.689909 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:02.689961 (+ 52us) server_negotiation.cc:407] Connection header received
0620 18:47:02.691815 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:02.691837 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:02.692170 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:02.692557 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:02.693601 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:02.693607 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:02.693619 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:47:02.693633 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:02.693706 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:47:02.693726 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:02.693956 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:02.694116 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 18:47:02.694305 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":326,"threads_started":1}
I20260620 18:47:02.698724 6022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:47:02.701805 27008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9a714c9104d45a7bdf688d7f3db0673: Bootstrap starting.
I20260620 18:47:02.705365 27008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9a714c9104d45a7bdf688d7f3db0673: Neither blocks nor log segments found. Creating new log.
I20260620 18:47:02.706610 27008 log.cc:824] T 00000000000000000000000000000000 P f9a714c9104d45a7bdf688d7f3db0673: Log is configured to *not* fsync() on all Append() calls
I20260620 18:47:02.712453 27008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9a714c9104d45a7bdf688d7f3db0673: No bootstrap required, opened a new log
I20260620 18:47:02.716691 27011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:02.714231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43595 (local address 127.0.0.1:37639)
0620 18:47:02.714429 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:02.714436 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:02.714769 (+ 333us) server_negotiation.cc:407] Connection header received
0620 18:47:02.714909 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:02.714914 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:02.714982 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:02.715094 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:02.715990 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:02.715996 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:02.716001 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:47:02.716019 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:02.716069 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:47:02.716090 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:02.716281 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:02.716336 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 18:47:02.716398 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:47:02.721071 27008 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9a714c9104d45a7bdf688d7f3db0673 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9a714c9104d45a7bdf688d7f3db0673" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37639 } }
I20260620 18:47:02.721443 27008 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9a714c9104d45a7bdf688d7f3db0673 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:47:02.721534 27008 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9a714c9104d45a7bdf688d7f3db0673 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9a714c9104d45a7bdf688d7f3db0673, State: Initialized, Role: FOLLOWER
I20260620 18:47:02.722153 27008 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9a714c9104d45a7bdf688d7f3db0673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9a714c9104d45a7bdf688d7f3db0673" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37639 } }
I20260620 18:47:02.722433 27008 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9a714c9104d45a7bdf688d7f3db0673 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:47:02.722569 27008 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9a714c9104d45a7bdf688d7f3db0673 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:47:02.722743 27008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9a714c9104d45a7bdf688d7f3db0673 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:47:02.724818 27008 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9a714c9104d45a7bdf688d7f3db0673 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9a714c9104d45a7bdf688d7f3db0673" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37639 } }
I20260620 18:47:02.725343 27008 leader_election.cc:302] T 00000000000000000000000000000000 P f9a714c9104d45a7bdf688d7f3db0673 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9a714c9104d45a7bdf688d7f3db0673; no voters:
I20260620 18:47:02.725911 27008 leader_election.cc:288] T 00000000000000000000000000000000 P f9a714c9104d45a7bdf688d7f3db0673 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:47:02.726263 27018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9a714c9104d45a7bdf688d7f3db0673 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:47:02.726959 27018 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9a714c9104d45a7bdf688d7f3db0673 [term 1 LEADER]: Becoming Leader. State: Replica: f9a714c9104d45a7bdf688d7f3db0673, State: Running, Role: LEADER
I20260620 18:47:02.727444 27018 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9a714c9104d45a7bdf688d7f3db0673 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9a714c9104d45a7bdf688d7f3db0673" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37639 } }
I20260620 18:47:02.727942 27008 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9a714c9104d45a7bdf688d7f3db0673 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:47:02.730089 27019 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9a714c9104d45a7bdf688d7f3db0673 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9a714c9104d45a7bdf688d7f3db0673" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9a714c9104d45a7bdf688d7f3db0673" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37639 } } }
I20260620 18:47:02.730029 27020 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9a714c9104d45a7bdf688d7f3db0673 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9a714c9104d45a7bdf688d7f3db0673. Latest consensus state: current_term: 1 leader_uuid: "f9a714c9104d45a7bdf688d7f3db0673" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9a714c9104d45a7bdf688d7f3db0673" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37639 } } }
I20260620 18:47:02.730643 27019 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9a714c9104d45a7bdf688d7f3db0673 [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:02.730643 27020 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9a714c9104d45a7bdf688d7f3db0673 [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:02.732329 27023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:47:02.736963 27023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:47:02.745644 27023 catalog_manager.cc:1347] Generated new cluster ID: 66c2a78da98944be87bca8c409dd1cd0
I20260620 18:47:02.745764 27023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:47:02.753830 27023 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:47:02.754299 27023 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:47:02.761523 27023 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9a714c9104d45a7bdf688d7f3db0673: Generated new TSK 0
I20260620 18:47:02.762558 27023 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:47:02.770936 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 26941
W20260620 18:47:02.771986 27003 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53977
W20260620 18:47:02.772668 6022 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 26941: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:47:02.772816 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 26941
2026-06-20T18:47:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1876 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:47:02 dist-test-slave-k6lg krb5kdc[27039](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:47:02 dist-test-slave-k6lg krb5kdc[27039](info): set up 2 sockets
Jun 20 18:47:02 dist-test-slave-k6lg krb5kdc[27039](info): commencing operation
krb5kdc: starting...
W20260620 18:47:03.938640 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.130s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:47:04 dist-test-slave-k6lg krb5kdc[27039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:47:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:47:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:47:04.183635 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34691
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:47:04.508260 27052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:04.508603 27052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:04.508776 27052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:04.515949 27052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:47:04.516111 27052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:04.516198 27052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:04.516287 27052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:47:04.516358 27052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:47:04.526144 27052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:47:04.527526 27052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:04.528863 27052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:04.540706 27057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:04.540649 27058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:04.540681 27060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:04.541488 27052 server_base.cc:1034] running on GCE node
I20260620 18:47:04.542316 27052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:04.543699 27052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:04.544898 27052 hybrid_clock.cc:648] HybridClock initialized: now 1781981224544860 us; error 25 us; skew 500 ppm
I20260620 18:47:04.545269 27052 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:04 dist-test-slave-k6lg krb5kdc[27039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:04.551450 27052 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:47:04.553049 27052 webserver.cc:460] Webserver started at http://127.0.0.1:36143/ using document root <none> and password file <none>
I20260620 18:47:04.553776 27052 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:04.553917 27052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:04.554301 27052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:47:04.557384 27052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "842a8716e9644ad1b2ccc97908b3b417"
format_stamp: "Formatted at 2026-06-20 18:47:04 on dist-test-slave-k6lg"
server_key: "0464997420ae745ed9fa9b79b27e7c70"
server_key_iv: "13a184cf28b1aed763e4427fcae868ff"
server_key_version: "encryptionkey@0"
I20260620 18:47:04.558223 27052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "842a8716e9644ad1b2ccc97908b3b417"
format_stamp: "Formatted at 2026-06-20 18:47:04 on dist-test-slave-k6lg"
server_key: "0464997420ae745ed9fa9b79b27e7c70"
server_key_iv: "13a184cf28b1aed763e4427fcae868ff"
server_key_version: "encryptionkey@0"
I20260620 18:47:04.563320 27052 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:47:04.566689 27067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:04.568022 27052 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 18:47:04.568221 27052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "842a8716e9644ad1b2ccc97908b3b417"
format_stamp: "Formatted at 2026-06-20 18:47:04 on dist-test-slave-k6lg"
server_key: "0464997420ae745ed9fa9b79b27e7c70"
server_key_iv: "13a184cf28b1aed763e4427fcae868ff"
server_key_version: "encryptionkey@0"
I20260620 18:47:04.568423 27052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:04.589807 27052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:04.593691 27052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:04.593972 27052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:04.612874 27052 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34691
I20260620 18:47:04.612918 27118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34691 every 8 connection(s)
I20260620 18:47:04.614286 27052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:47:04.616415 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 27052
I20260620 18:47:04.616643 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:47:04.617256 6022 external_mini_cluster.cc:1428] Setting key 2e4eb35e0a845e74f3d0b1539854565a
I20260620 18:47:04.619172 27119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 18:47:04 dist-test-slave-k6lg krb5kdc[27039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 18:47:04.633842 27119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 842a8716e9644ad1b2ccc97908b3b417: Bootstrap starting.
I20260620 18:47:04.637284 27119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 842a8716e9644ad1b2ccc97908b3b417: Neither blocks nor log segments found. Creating new log.
I20260620 18:47:04.638460 27119 log.cc:824] T 00000000000000000000000000000000 P 842a8716e9644ad1b2ccc97908b3b417: Log is configured to *not* fsync() on all Append() calls
I20260620 18:47:04.641588 27119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 842a8716e9644ad1b2ccc97908b3b417: No bootstrap required, opened a new log
I20260620 18:47:04.641608 27122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:04.619092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:51489 (local address 127.0.0.1:34691)
0620 18:47:04.619921 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:04.619934 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:04.619974 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:47:04.621658 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:04.621681 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:04.622088 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:04.622497 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:04.624471 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:04.625802 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:04.627802 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:04.628986 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:04.629206 (+ 220us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:04.634368 (+ 5162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:04.634376 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:04.634390 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:04.634425 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:04.637458 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:04.638277 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:04.638284 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:04.638289 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:04.638459 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:04.638932 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:04.638935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:04.638938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:04.639480 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:04.639802 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:04.640337 (+ 535us) server_negotiation.cc:299] Negotiation successful
0620 18:47:04.640639 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":385,"threads_started":1}
I20260620 18:47:04.645692 6022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:47:04.651090 27119 raft_consensus.cc:348] T 00000000000000000000000000000000 P 842a8716e9644ad1b2ccc97908b3b417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842a8716e9644ad1b2ccc97908b3b417" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34691 } }
I20260620 18:47:04.651542 27119 raft_consensus.cc:374] T 00000000000000000000000000000000 P 842a8716e9644ad1b2ccc97908b3b417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:47:04.651667 27119 raft_consensus.cc:729] T 00000000000000000000000000000000 P 842a8716e9644ad1b2ccc97908b3b417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842a8716e9644ad1b2ccc97908b3b417, State: Initialized, Role: FOLLOWER
I20260620 18:47:04.652254 27119 consensus_queue.cc:260] T 00000000000000000000000000000000 P 842a8716e9644ad1b2ccc97908b3b417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842a8716e9644ad1b2ccc97908b3b417" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34691 } }
I20260620 18:47:04.652510 27119 raft_consensus.cc:388] T 00000000000000000000000000000000 P 842a8716e9644ad1b2ccc97908b3b417 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:47:04.652694 27119 raft_consensus.cc:482] T 00000000000000000000000000000000 P 842a8716e9644ad1b2ccc97908b3b417 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:47:04.652915 27119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 842a8716e9644ad1b2ccc97908b3b417 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:47:04.655428 27119 raft_consensus.cc:504] T 00000000000000000000000000000000 P 842a8716e9644ad1b2ccc97908b3b417 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842a8716e9644ad1b2ccc97908b3b417" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34691 } }
I20260620 18:47:04.656010 27119 leader_election.cc:302] T 00000000000000000000000000000000 P 842a8716e9644ad1b2ccc97908b3b417 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 842a8716e9644ad1b2ccc97908b3b417; no voters:
I20260620 18:47:04.656708 27119 leader_election.cc:288] T 00000000000000000000000000000000 P 842a8716e9644ad1b2ccc97908b3b417 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:47:04.656818 27129 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 842a8716e9644ad1b2ccc97908b3b417 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:47:04.658046 27129 raft_consensus.cc:686] T 00000000000000000000000000000000 P 842a8716e9644ad1b2ccc97908b3b417 [term 1 LEADER]: Becoming Leader. State: Replica: 842a8716e9644ad1b2ccc97908b3b417, State: Running, Role: LEADER
I20260620 18:47:04.658769 27129 consensus_queue.cc:237] T 00000000000000000000000000000000 P 842a8716e9644ad1b2ccc97908b3b417 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842a8716e9644ad1b2ccc97908b3b417" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34691 } }
I20260620 18:47:04.659452 27119 sys_catalog.cc:564] T 00000000000000000000000000000000 P 842a8716e9644ad1b2ccc97908b3b417 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:47:04.661907 27131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 842a8716e9644ad1b2ccc97908b3b417 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 842a8716e9644ad1b2ccc97908b3b417. Latest consensus state: current_term: 1 leader_uuid: "842a8716e9644ad1b2ccc97908b3b417" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842a8716e9644ad1b2ccc97908b3b417" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34691 } } }
I20260620 18:47:04.662331 27131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 842a8716e9644ad1b2ccc97908b3b417 [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:04.662011 27130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 842a8716e9644ad1b2ccc97908b3b417 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "842a8716e9644ad1b2ccc97908b3b417" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842a8716e9644ad1b2ccc97908b3b417" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34691 } } }
I20260620 18:47:04.662757 27130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 842a8716e9644ad1b2ccc97908b3b417 [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:04.667024 27122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:04.650026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:33265 (local address 127.0.0.1:34691)
0620 18:47:04.650250 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:04.650257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:04.650651 (+ 394us) server_negotiation.cc:407] Connection header received
0620 18:47:04.650809 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:04.650815 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:04.650905 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:04.651014 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:04.652684 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:04.653549 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:04.654864 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:04.655301 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:04.655446 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:04.658381 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:04.658386 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:04.658389 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:04.658420 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:04.662284 (+ 3864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:04.663497 (+ 1213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:04.663507 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:04.663514 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:04.663755 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:04.664485 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:04.664493 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:04.664500 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:04.665000 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:04.665368 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:04.666258 (+ 890us) server_negotiation.cc:299] Negotiation successful
0620 18:47:04.666501 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:47:04.670850 27135 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:47:04.676312 27135 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:47:04.685652 27135 catalog_manager.cc:1347] Generated new cluster ID: 59f2a7bdd7e2492fa71de8e49fcae3fe
I20260620 18:47:04.685783 27135 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:47:04.699458 27135 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:47:04.701296 27135 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:47:04.717268 27135 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 842a8716e9644ad1b2ccc97908b3b417: Generated new TSK 0
I20260620 18:47:04.718398 27135 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:47:04.725229 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 27052
I20260620 18:47:04.726678 27114 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:47:04.820696 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 27052
W20260620 18:47:04.837929 27037 connection.cc:535] client connection to 127.0.0.1:34691 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34691): Connection reset by peer (error 104)
2026-06-20T18:47:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2049 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:47:04 dist-test-slave-k6lg krb5kdc[27151](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:47:04 dist-test-slave-k6lg krb5kdc[27151](info): set up 2 sockets
Jun 20 18:47:04 dist-test-slave-k6lg krb5kdc[27151](info): commencing operation
krb5kdc: starting...
W20260620 18:47:05.930646 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.007s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:47:06 dist-test-slave-k6lg krb5kdc[27151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:47:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:47:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:47:06.179611 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38033
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:47:06.501708 27164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:06.502048 27164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:06.502243 27164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:06.509378 27164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:47:06.509536 27164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:06.509636 27164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:06.509740 27164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:47:06.509820 27164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:47:06.520015 27164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:47:06.521323 27164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:06.522643 27164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:06.534557 27172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:06.534638 27170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:06.534601 27169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:06.535399 27164 server_base.cc:1034] running on GCE node
I20260620 18:47:06.536181 27164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:06.537616 27164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:06.538849 27164 hybrid_clock.cc:648] HybridClock initialized: now 1781981226538808 us; error 36 us; skew 500 ppm
I20260620 18:47:06.539224 27164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:06 dist-test-slave-k6lg krb5kdc[27151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:06.542332 27164 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:47:06.543864 27164 webserver.cc:460] Webserver started at http://127.0.0.1:45007/ using document root <none> and password file <none>
I20260620 18:47:06.544526 27164 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:06.544651 27164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:06.544947 27164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:47:06.547868 27164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "12562ad20e864830a4cf0e5ae7ea2f72"
format_stamp: "Formatted at 2026-06-20 18:47:06 on dist-test-slave-k6lg"
server_key: "63db428c6af40e7c84086c47d4aab1ca"
server_key_iv: "6c818ac43d98bf9af3ad2ae866d9cba0"
server_key_version: "encryptionkey@0"
I20260620 18:47:06.548604 27164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "12562ad20e864830a4cf0e5ae7ea2f72"
format_stamp: "Formatted at 2026-06-20 18:47:06 on dist-test-slave-k6lg"
server_key: "63db428c6af40e7c84086c47d4aab1ca"
server_key_iv: "6c818ac43d98bf9af3ad2ae866d9cba0"
server_key_version: "encryptionkey@0"
I20260620 18:47:06.553560 27164 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:47:06.556888 27179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:06.558223 27164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 18:47:06.558398 27164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "12562ad20e864830a4cf0e5ae7ea2f72"
format_stamp: "Formatted at 2026-06-20 18:47:06 on dist-test-slave-k6lg"
server_key: "63db428c6af40e7c84086c47d4aab1ca"
server_key_iv: "6c818ac43d98bf9af3ad2ae866d9cba0"
server_key_version: "encryptionkey@0"
I20260620 18:47:06.558585 27164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:06.587260 27164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:06.591300 27164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:06.591583 27164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:06.610021 27164 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38033
I20260620 18:47:06.610046 27230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38033 every 8 connection(s)
I20260620 18:47:06.611464 27164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:47:06.612095 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 27164
I20260620 18:47:06.612310 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:47:06.612867 6022 external_mini_cluster.cc:1428] Setting key 49f168a640de2456ae22466dfe809be0
I20260620 18:47:06.615779 27231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:47:06.625526 27234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:06.614700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:46577 (local address 127.0.0.1:38033)
0620 18:47:06.615459 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:06.615471 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:06.615511 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:47:06.617205 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:06.617226 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:06.617602 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:06.618004 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:06.618609 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:06.619902 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:06.621185 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:06.621845 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:06.622033 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:06.623320 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:06.623328 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:06.623341 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:47:06.623904 (+ 563us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:06.623980 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:47:06.624003 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:06.624184 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:06.624468 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 18:47:06.624624 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":317,"threads_started":1}
I20260620 18:47:06.626520 27231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12562ad20e864830a4cf0e5ae7ea2f72: Bootstrap starting.
I20260620 18:47:06.629480 6022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:47:06.630142 27231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12562ad20e864830a4cf0e5ae7ea2f72: Neither blocks nor log segments found. Creating new log.
I20260620 18:47:06.631158 27231 log.cc:824] T 00000000000000000000000000000000 P 12562ad20e864830a4cf0e5ae7ea2f72: Log is configured to *not* fsync() on all Append() calls
I20260620 18:47:06.634366 27231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12562ad20e864830a4cf0e5ae7ea2f72: No bootstrap required, opened a new log
I20260620 18:47:06.640033 27234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:06.633847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:39665 (local address 127.0.0.1:38033)
0620 18:47:06.634004 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:06.634009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:06.634559 (+ 550us) server_negotiation.cc:407] Connection header received
0620 18:47:06.634749 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:06.634758 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:06.634823 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:06.634928 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:06.635452 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:06.636235 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:06.637533 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:06.637972 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:06.638117 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:06.639278 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:06.639282 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:06.639286 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:47:06.639305 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:06.639361 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:47:06.639382 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:06.639514 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:06.639696 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 18:47:06.639755 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 18:47:06.642815 27231 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12562ad20e864830a4cf0e5ae7ea2f72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12562ad20e864830a4cf0e5ae7ea2f72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38033 } }
I20260620 18:47:06.643181 27231 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12562ad20e864830a4cf0e5ae7ea2f72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:47:06.643289 27231 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12562ad20e864830a4cf0e5ae7ea2f72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12562ad20e864830a4cf0e5ae7ea2f72, State: Initialized, Role: FOLLOWER
I20260620 18:47:06.643847 27231 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12562ad20e864830a4cf0e5ae7ea2f72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12562ad20e864830a4cf0e5ae7ea2f72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38033 } }
I20260620 18:47:06.644073 27231 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12562ad20e864830a4cf0e5ae7ea2f72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:47:06.644217 27231 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12562ad20e864830a4cf0e5ae7ea2f72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:47:06.644385 27231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12562ad20e864830a4cf0e5ae7ea2f72 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:47:06.646661 27231 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12562ad20e864830a4cf0e5ae7ea2f72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12562ad20e864830a4cf0e5ae7ea2f72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38033 } }
I20260620 18:47:06.647150 27231 leader_election.cc:302] T 00000000000000000000000000000000 P 12562ad20e864830a4cf0e5ae7ea2f72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12562ad20e864830a4cf0e5ae7ea2f72; no voters:
I20260620 18:47:06.647828 27231 leader_election.cc:288] T 00000000000000000000000000000000 P 12562ad20e864830a4cf0e5ae7ea2f72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:47:06.647953 27241 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12562ad20e864830a4cf0e5ae7ea2f72 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:47:06.648654 27241 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12562ad20e864830a4cf0e5ae7ea2f72 [term 1 LEADER]: Becoming Leader. State: Replica: 12562ad20e864830a4cf0e5ae7ea2f72, State: Running, Role: LEADER
I20260620 18:47:06.649209 27241 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12562ad20e864830a4cf0e5ae7ea2f72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12562ad20e864830a4cf0e5ae7ea2f72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38033 } }
I20260620 18:47:06.649890 27231 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12562ad20e864830a4cf0e5ae7ea2f72 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:47:06.652114 27243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12562ad20e864830a4cf0e5ae7ea2f72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12562ad20e864830a4cf0e5ae7ea2f72. Latest consensus state: current_term: 1 leader_uuid: "12562ad20e864830a4cf0e5ae7ea2f72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12562ad20e864830a4cf0e5ae7ea2f72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38033 } } }
I20260620 18:47:06.652518 27243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12562ad20e864830a4cf0e5ae7ea2f72 [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:06.652999 27242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12562ad20e864830a4cf0e5ae7ea2f72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12562ad20e864830a4cf0e5ae7ea2f72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12562ad20e864830a4cf0e5ae7ea2f72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38033 } } }
I20260620 18:47:06.653280 27242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12562ad20e864830a4cf0e5ae7ea2f72 [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:06.657913 27248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:47:06.663350 27248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:47:06.672665 27248 catalog_manager.cc:1347] Generated new cluster ID: f3efeb9e8e504ae6bcc00669df1003b4
I20260620 18:47:06.672791 27248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:47:06.686367 27248 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:47:06.687600 27248 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:47:06.694260 27248 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12562ad20e864830a4cf0e5ae7ea2f72: Generated new TSK 0
I20260620 18:47:06.694981 27248 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:47:06.706437 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 27164
W20260620 18:47:06.707453 27226 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.164:46577
W20260620 18:47:06.708146 6022 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 27164: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:47:06.708295 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 27164
W20260620 18:47:06.725221 27149 connection.cc:535] client connection to 127.0.0.1:38033 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38033): Connection reset by peer (error 104)
2026-06-20T18:47:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1886 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:47:06 dist-test-slave-k6lg krb5kdc[27262](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:47:06 dist-test-slave-k6lg krb5kdc[27262](info): set up 2 sockets
Jun 20 18:47:06 dist-test-slave-k6lg krb5kdc[27262](info): commencing operation
krb5kdc: starting...
W20260620 18:47:07.814874 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:47:07 dist-test-slave-k6lg krb5kdc[27262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:47:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:47:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:47:08.071761 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41859
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:47:08.398048 27275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:08.398352 27275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:08.398536 27275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:08.405890 27275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:47:08.406003 27275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:08.406116 27275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:08.406235 27275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:47:08.406309 27275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:47:08.416231 27275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:47:08.417569 27275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:08.418926 27275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:08.431013 27281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:08.431087 27280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:08.430987 27283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:08.431708 27275 server_base.cc:1034] running on GCE node
I20260620 18:47:08.432508 27275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:08.433900 27275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:08.435144 27275 hybrid_clock.cc:648] HybridClock initialized: now 1781981228435097 us; error 41 us; skew 500 ppm
I20260620 18:47:08.435503 27275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:08 dist-test-slave-k6lg krb5kdc[27262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:08.438680 27275 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:47:08.440186 27275 webserver.cc:460] Webserver started at http://127.0.0.1:46535/ using document root <none> and password file <none>
I20260620 18:47:08.440879 27275 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:08.441004 27275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:08.441315 27275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:47:08.444432 27275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "4442988b2cea47c79f832695b2fa6e25"
format_stamp: "Formatted at 2026-06-20 18:47:08 on dist-test-slave-k6lg"
server_key: "c95719e7ee521ed3627476f9e9abf991"
server_key_iv: "df64d419bfa8ba41ff84978344bf71d1"
server_key_version: "encryptionkey@0"
I20260620 18:47:08.445204 27275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "4442988b2cea47c79f832695b2fa6e25"
format_stamp: "Formatted at 2026-06-20 18:47:08 on dist-test-slave-k6lg"
server_key: "c95719e7ee521ed3627476f9e9abf991"
server_key_iv: "df64d419bfa8ba41ff84978344bf71d1"
server_key_version: "encryptionkey@0"
I20260620 18:47:08.450385 27275 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 18:47:08.453765 27290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:08.455052 27275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:47:08.455219 27275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "4442988b2cea47c79f832695b2fa6e25"
format_stamp: "Formatted at 2026-06-20 18:47:08 on dist-test-slave-k6lg"
server_key: "c95719e7ee521ed3627476f9e9abf991"
server_key_iv: "df64d419bfa8ba41ff84978344bf71d1"
server_key_version: "encryptionkey@0"
I20260620 18:47:08.455400 27275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:08.461606 27275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:08.465646 27275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:08.465942 27275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:08.483736 27275 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41859
I20260620 18:47:08.483778 27341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41859 every 8 connection(s)
I20260620 18:47:08.485148 27275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:47:08.488940 27342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:47:08.495227 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 27275
I20260620 18:47:08.495417 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:47:08.495837 6022 external_mini_cluster.cc:1428] Setting key e37d33cdc47834f9485e5cd3c381d3bb
I20260620 18:47:08.500361 27342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4442988b2cea47c79f832695b2fa6e25: Bootstrap starting.
I20260620 18:47:08.504621 27342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4442988b2cea47c79f832695b2fa6e25: Neither blocks nor log segments found. Creating new log.
I20260620 18:47:08.504976 27345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:08.497631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:35539 (local address 127.0.0.1:41859)
0620 18:47:08.498461 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:08.498477 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:08.498526 (+ 49us) server_negotiation.cc:407] Connection header received
0620 18:47:08.500885 (+ 2359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:08.500920 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:08.501371 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:08.501801 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:08.503014 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:08.503022 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:08.503039 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:47:08.503101 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:08.503178 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:47:08.503204 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:08.503421 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:08.503544 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 18:47:08.503751 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":328,"threads_started":1}
I20260620 18:47:08.505833 27342 log.cc:824] T 00000000000000000000000000000000 P 4442988b2cea47c79f832695b2fa6e25: Log is configured to *not* fsync() on all Append() calls
I20260620 18:47:08.508971 6022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:47:08.509105 27342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4442988b2cea47c79f832695b2fa6e25: No bootstrap required, opened a new log
I20260620 18:47:08.516314 27345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:08.513403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:36309 (local address 127.0.0.1:41859)
0620 18:47:08.513565 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:08.513569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:08.514031 (+ 462us) server_negotiation.cc:407] Connection header received
0620 18:47:08.514232 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:08.514238 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:08.514305 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:08.514412 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:08.515562 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:08.515568 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:08.515573 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:47:08.515592 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:08.515643 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:47:08.515662 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:08.515825 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:08.515874 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 18:47:08.515943 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 18:47:08.518536 27342 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4442988b2cea47c79f832695b2fa6e25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4442988b2cea47c79f832695b2fa6e25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41859 } }
I20260620 18:47:08.518903 27342 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4442988b2cea47c79f832695b2fa6e25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:47:08.518975 27342 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4442988b2cea47c79f832695b2fa6e25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4442988b2cea47c79f832695b2fa6e25, State: Initialized, Role: FOLLOWER
I20260620 18:47:08.519479 27342 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4442988b2cea47c79f832695b2fa6e25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4442988b2cea47c79f832695b2fa6e25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41859 } }
I20260620 18:47:08.519709 27342 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4442988b2cea47c79f832695b2fa6e25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:47:08.519804 27342 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4442988b2cea47c79f832695b2fa6e25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:47:08.519927 27342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4442988b2cea47c79f832695b2fa6e25 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:47:08.522138 27342 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4442988b2cea47c79f832695b2fa6e25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4442988b2cea47c79f832695b2fa6e25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41859 } }
I20260620 18:47:08.522769 27342 leader_election.cc:302] T 00000000000000000000000000000000 P 4442988b2cea47c79f832695b2fa6e25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4442988b2cea47c79f832695b2fa6e25; no voters:
I20260620 18:47:08.523466 27342 leader_election.cc:288] T 00000000000000000000000000000000 P 4442988b2cea47c79f832695b2fa6e25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:47:08.523540 27352 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4442988b2cea47c79f832695b2fa6e25 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:47:08.524307 27352 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4442988b2cea47c79f832695b2fa6e25 [term 1 LEADER]: Becoming Leader. State: Replica: 4442988b2cea47c79f832695b2fa6e25, State: Running, Role: LEADER
I20260620 18:47:08.524900 27352 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4442988b2cea47c79f832695b2fa6e25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4442988b2cea47c79f832695b2fa6e25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41859 } }
I20260620 18:47:08.525719 27342 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4442988b2cea47c79f832695b2fa6e25 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:47:08.527673 27354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4442988b2cea47c79f832695b2fa6e25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4442988b2cea47c79f832695b2fa6e25. Latest consensus state: current_term: 1 leader_uuid: "4442988b2cea47c79f832695b2fa6e25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4442988b2cea47c79f832695b2fa6e25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41859 } } }
I20260620 18:47:08.528175 27354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4442988b2cea47c79f832695b2fa6e25 [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:08.528673 27353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4442988b2cea47c79f832695b2fa6e25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4442988b2cea47c79f832695b2fa6e25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4442988b2cea47c79f832695b2fa6e25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41859 } } }
I20260620 18:47:08.529013 27353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4442988b2cea47c79f832695b2fa6e25 [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:08.530396 27361 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:47:08.537883 27361 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:47:08.547554 27361 catalog_manager.cc:1347] Generated new cluster ID: 4df55ab4183a4fb8b11877fa3a2b10fb
I20260620 18:47:08.547700 27361 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:47:08.577740 27361 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:47:08.578250 27361 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:47:08.587930 27361 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4442988b2cea47c79f832695b2fa6e25: Generated new TSK 0
I20260620 18:47:08.588780 27361 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:47:08.596899 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 27275
W20260620 18:47:08.597965 27337 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.164:35539
W20260620 18:47:08.598686 6022 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 27275: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:47:08.598836 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 27275
2026-06-20T18:47:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1891 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:47:08 dist-test-slave-k6lg krb5kdc[27373](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:47:08 dist-test-slave-k6lg krb5kdc[27373](info): set up 2 sockets
Jun 20 18:47:08 dist-test-slave-k6lg krb5kdc[27373](info): commencing operation
krb5kdc: starting...
W20260620 18:47:09.710722 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:47:09 dist-test-slave-k6lg krb5kdc[27373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:47:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:47:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:47:09.957167 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44379
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:47:10.281378 27386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:10.281720 27386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:10.281890 27386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:10.289103 27386 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:47:10.289255 27386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:10.289351 27386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:10.289454 27386 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:47:10.289532 27386 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:47:10.299316 27386 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44379
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:47:10.300704 27386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:10.302001 27386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:10.313939 27391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:10.313930 27392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:10.313985 27394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:10.314788 27386 server_base.cc:1034] running on GCE node
I20260620 18:47:10.315548 27386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:10.317031 27386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:10.318282 27386 hybrid_clock.cc:648] HybridClock initialized: now 1781981230318211 us; error 56 us; skew 500 ppm
I20260620 18:47:10.318647 27386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:10 dist-test-slave-k6lg krb5kdc[27373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:10.326308 27386 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:47:10.327904 27386 webserver.cc:460] Webserver started at http://127.0.0.1:38261/ using document root <none> and password file <none>
I20260620 18:47:10.328660 27386 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:10.328799 27386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:10.329124 27386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:47:10.332242 27386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "0354c548421e4f69b5d9654e03972284"
format_stamp: "Formatted at 2026-06-20 18:47:10 on dist-test-slave-k6lg"
server_key: "98f470d1e1c15b5baef23cbbcbe97d69"
server_key_iv: "755bf5eb69e328a916d9474f4484cef9"
server_key_version: "encryptionkey@0"
I20260620 18:47:10.333022 27386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "0354c548421e4f69b5d9654e03972284"
format_stamp: "Formatted at 2026-06-20 18:47:10 on dist-test-slave-k6lg"
server_key: "98f470d1e1c15b5baef23cbbcbe97d69"
server_key_iv: "755bf5eb69e328a916d9474f4484cef9"
server_key_version: "encryptionkey@0"
I20260620 18:47:10.339670 27386 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260620 18:47:10.343235 27401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:10.344591 27386 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:47:10.344784 27386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "0354c548421e4f69b5d9654e03972284"
format_stamp: "Formatted at 2026-06-20 18:47:10 on dist-test-slave-k6lg"
server_key: "98f470d1e1c15b5baef23cbbcbe97d69"
server_key_iv: "755bf5eb69e328a916d9474f4484cef9"
server_key_version: "encryptionkey@0"
I20260620 18:47:10.344991 27386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:10.363314 27386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:10.367342 27386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:10.367622 27386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:10.386132 27386 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44379
I20260620 18:47:10.386150 27452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44379 every 8 connection(s)
I20260620 18:47:10.387534 27386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:47:10.389952 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 27386
I20260620 18:47:10.390132 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:47:10.390583 6022 external_mini_cluster.cc:1428] Setting key b2de5afbcbeb717184d81691e1c35743
I20260620 18:47:10.391914 27453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:47:10.403604 27453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0354c548421e4f69b5d9654e03972284: Bootstrap starting.
Jun 20 18:47:10 dist-test-slave-k6lg krb5kdc[27373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 18:47:10.407316 27453 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0354c548421e4f69b5d9654e03972284: Neither blocks nor log segments found. Creating new log.
I20260620 18:47:10.408499 27453 log.cc:824] T 00000000000000000000000000000000 P 0354c548421e4f69b5d9654e03972284: Log is configured to *not* fsync() on all Append() calls
I20260620 18:47:10.411876 27453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0354c548421e4f69b5d9654e03972284: No bootstrap required, opened a new log
I20260620 18:47:10.412793 27456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:10.392379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:41693 (local address 127.0.0.1:44379)
0620 18:47:10.393154 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:10.393166 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:10.393205 (+ 39us) server_negotiation.cc:407] Connection header received
0620 18:47:10.394915 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:10.394934 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:10.395338 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:10.395727 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:10.397160 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:10.398509 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:10.399794 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:10.400484 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:10.400694 (+ 210us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:10.405220 (+ 4526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:10.405228 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:10.405241 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:10.405278 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:10.408451 (+ 3173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:10.409436 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:10.409446 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:10.409457 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:10.409685 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:10.410261 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:10.410265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:10.410267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:10.410746 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:10.410976 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:10.411493 (+ 517us) server_negotiation.cc:299] Negotiation successful
0620 18:47:10.411772 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":323,"threads_started":1}
I20260620 18:47:10.416522 6022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:47:10.421288 27453 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0354c548421e4f69b5d9654e03972284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0354c548421e4f69b5d9654e03972284" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44379 } }
I20260620 18:47:10.421742 27453 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0354c548421e4f69b5d9654e03972284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:47:10.421912 27453 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0354c548421e4f69b5d9654e03972284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0354c548421e4f69b5d9654e03972284, State: Initialized, Role: FOLLOWER
I20260620 18:47:10.422626 27453 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0354c548421e4f69b5d9654e03972284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0354c548421e4f69b5d9654e03972284" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44379 } }
I20260620 18:47:10.422863 27453 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0354c548421e4f69b5d9654e03972284 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:47:10.422996 27453 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0354c548421e4f69b5d9654e03972284 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:47:10.423250 27453 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0354c548421e4f69b5d9654e03972284 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:47:10.425386 27453 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0354c548421e4f69b5d9654e03972284 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0354c548421e4f69b5d9654e03972284" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44379 } }
I20260620 18:47:10.425863 27453 leader_election.cc:302] T 00000000000000000000000000000000 P 0354c548421e4f69b5d9654e03972284 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0354c548421e4f69b5d9654e03972284; no voters:
I20260620 18:47:10.426537 27453 leader_election.cc:288] T 00000000000000000000000000000000 P 0354c548421e4f69b5d9654e03972284 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:47:10.427493 27463 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0354c548421e4f69b5d9654e03972284 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:47:10.429760 27463 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0354c548421e4f69b5d9654e03972284 [term 1 LEADER]: Becoming Leader. State: Replica: 0354c548421e4f69b5d9654e03972284, State: Running, Role: LEADER
I20260620 18:47:10.430436 27453 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0354c548421e4f69b5d9654e03972284 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:47:10.430487 27463 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0354c548421e4f69b5d9654e03972284 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0354c548421e4f69b5d9654e03972284" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44379 } }
I20260620 18:47:10.435684 27465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0354c548421e4f69b5d9654e03972284 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0354c548421e4f69b5d9654e03972284. Latest consensus state: current_term: 1 leader_uuid: "0354c548421e4f69b5d9654e03972284" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0354c548421e4f69b5d9654e03972284" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44379 } } }
I20260620 18:47:10.436050 27465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0354c548421e4f69b5d9654e03972284 [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:10.436607 27464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0354c548421e4f69b5d9654e03972284 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0354c548421e4f69b5d9654e03972284" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0354c548421e4f69b5d9654e03972284" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44379 } } }
I20260620 18:47:10.436933 27464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0354c548421e4f69b5d9654e03972284 [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:10.437129 27470 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:47:10.442131 27456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:10.421732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:51981 (local address 127.0.0.1:44379)
0620 18:47:10.421914 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:10.421919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:10.422307 (+ 388us) server_negotiation.cc:407] Connection header received
0620 18:47:10.422488 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:10.422498 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:10.422556 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:10.422659 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:10.424178 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:10.424947 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:10.426402 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:10.427187 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:10.427314 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:10.429776 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:10.429784 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:10.429789 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:10.429833 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:10.434179 (+ 4346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:10.436824 (+ 2645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:10.436831 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:10.436840 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:10.437022 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:10.437654 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:10.437658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:10.437661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:10.439467 (+ 1806us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:10.439840 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:10.441323 (+ 1483us) server_negotiation.cc:299] Negotiation successful
0620 18:47:10.441579 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:47:10.447907 27470 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:47:10.458781 27470 catalog_manager.cc:1347] Generated new cluster ID: c78f095bbd1b40448c02fbf9af5af212
I20260620 18:47:10.458900 27470 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:47:10.471570 27470 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:47:10.472990 27470 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:47:10.483465 27470 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0354c548421e4f69b5d9654e03972284: Generated new TSK 0
I20260620 18:47:10.484277 27470 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:47:10.492717 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 27386
I20260620 18:47:10.494553 27448 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:47:10.586776 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 27386
W20260620 18:47:10.604344 27371 connection.cc:535] client connection to 127.0.0.1:44379 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44379): Connection reset by peer (error 104)
2026-06-20T18:47:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1987 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:47:10 dist-test-slave-k6lg krb5kdc[27485](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:47:10 dist-test-slave-k6lg krb5kdc[27485](info): set up 2 sockets
Jun 20 18:47:10 dist-test-slave-k6lg krb5kdc[27485](info): commencing operation
krb5kdc: starting...
W20260620 18:47:11.694802 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:47:11 dist-test-slave-k6lg krb5kdc[27485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:47:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:47:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:47:11.957813 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38855
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:47:12.290390 27498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:12.290716 27498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:12.290880 27498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:12.297967 27498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:47:12.298121 27498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:12.298245 27498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:12.298353 27498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:47:12.298439 27498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:47:12.308137 27498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38855
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:47:12.309458 27498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:12.310859 27498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:12.323485 27503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:12.323428 27506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:12.323460 27504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:12.324337 27498 server_base.cc:1034] running on GCE node
I20260620 18:47:12.325079 27498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:12.326524 27498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:12.327755 27498 hybrid_clock.cc:648] HybridClock initialized: now 1781981232327710 us; error 45 us; skew 500 ppm
I20260620 18:47:12.328128 27498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:12 dist-test-slave-k6lg krb5kdc[27485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:12.331348 27498 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:47:12.332772 27498 webserver.cc:460] Webserver started at http://127.0.0.1:38893/ using document root <none> and password file <none>
I20260620 18:47:12.333410 27498 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:12.333518 27498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:12.333765 27498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:47:12.336741 27498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "1bb6de7c4f944007b1a42ba54b1b5581"
format_stamp: "Formatted at 2026-06-20 18:47:12 on dist-test-slave-k6lg"
server_key: "9722fe0bd1380ee6dd50dd05c3ff5f7a"
server_key_iv: "8f983ee005aaacc88b20d1985174634b"
server_key_version: "encryptionkey@0"
I20260620 18:47:12.337450 27498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "1bb6de7c4f944007b1a42ba54b1b5581"
format_stamp: "Formatted at 2026-06-20 18:47:12 on dist-test-slave-k6lg"
server_key: "9722fe0bd1380ee6dd50dd05c3ff5f7a"
server_key_iv: "8f983ee005aaacc88b20d1985174634b"
server_key_version: "encryptionkey@0"
I20260620 18:47:12.342723 27498 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:47:12.347035 27513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:12.348659 27498 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 18:47:12.348801 27498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "1bb6de7c4f944007b1a42ba54b1b5581"
format_stamp: "Formatted at 2026-06-20 18:47:12 on dist-test-slave-k6lg"
server_key: "9722fe0bd1380ee6dd50dd05c3ff5f7a"
server_key_iv: "8f983ee005aaacc88b20d1985174634b"
server_key_version: "encryptionkey@0"
I20260620 18:47:12.348976 27498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:12.361230 27498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:12.365293 27498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:12.365538 27498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:12.384327 27498 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38855
I20260620 18:47:12.384346 27564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38855 every 8 connection(s)
I20260620 18:47:12.385795 27498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:47:12.389788 27565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:47:12.392313 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 27498
I20260620 18:47:12.392490 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:47:12.393044 6022 external_mini_cluster.cc:1428] Setting key bd08d421fb1224ccf77af72fe9d57550
I20260620 18:47:12.401892 27565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bb6de7c4f944007b1a42ba54b1b5581: Bootstrap starting.
I20260620 18:47:12.406145 27565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bb6de7c4f944007b1a42ba54b1b5581: Neither blocks nor log segments found. Creating new log.
I20260620 18:47:12.407466 27568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:12.395051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:57479 (local address 127.0.0.1:38855)
0620 18:47:12.396024 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:12.396041 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:12.396086 (+ 45us) server_negotiation.cc:407] Connection header received
0620 18:47:12.397960 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:12.397984 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:12.398512 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:12.398921 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:12.399482 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:12.400793 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:12.402153 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:12.403195 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:12.403388 (+ 193us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:12.404612 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:12.404623 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:12.404640 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:47:12.405355 (+ 715us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:12.405448 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:47:12.405480 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:12.405639 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:12.405886 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 18:47:12.406073 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":365,"threads_started":1}
I20260620 18:47:12.407521 27565 log.cc:824] T 00000000000000000000000000000000 P 1bb6de7c4f944007b1a42ba54b1b5581: Log is configured to *not* fsync() on all Append() calls
I20260620 18:47:12.411074 27565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bb6de7c4f944007b1a42ba54b1b5581: No bootstrap required, opened a new log
I20260620 18:47:12.411612 6022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:47:12.420197 27565 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bb6de7c4f944007b1a42ba54b1b5581 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb6de7c4f944007b1a42ba54b1b5581" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38855 } }
I20260620 18:47:12.420706 27565 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bb6de7c4f944007b1a42ba54b1b5581 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:47:12.420827 27565 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bb6de7c4f944007b1a42ba54b1b5581 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bb6de7c4f944007b1a42ba54b1b5581, State: Initialized, Role: FOLLOWER
I20260620 18:47:12.421558 27565 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bb6de7c4f944007b1a42ba54b1b5581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb6de7c4f944007b1a42ba54b1b5581" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38855 } }
I20260620 18:47:12.421845 27565 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bb6de7c4f944007b1a42ba54b1b5581 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:47:12.421993 27565 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bb6de7c4f944007b1a42ba54b1b5581 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:47:12.422258 27565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bb6de7c4f944007b1a42ba54b1b5581 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:47:12.423260 27568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:12.416025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:58453 (local address 127.0.0.1:38855)
0620 18:47:12.416254 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:12.416264 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:12.416702 (+ 438us) server_negotiation.cc:407] Connection header received
0620 18:47:12.416883 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:12.416887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:12.416953 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:12.417036 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:12.417653 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:12.418561 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:12.420411 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:12.420938 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:12.421106 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:12.422333 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:12.422339 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:12.422344 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:47:12.422381 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:12.422449 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:47:12.422475 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:12.422607 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:12.422864 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 18:47:12.422924 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:47:12.424901 27565 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bb6de7c4f944007b1a42ba54b1b5581 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb6de7c4f944007b1a42ba54b1b5581" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38855 } }
I20260620 18:47:12.425417 27565 leader_election.cc:302] T 00000000000000000000000000000000 P 1bb6de7c4f944007b1a42ba54b1b5581 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bb6de7c4f944007b1a42ba54b1b5581; no voters:
I20260620 18:47:12.426008 27565 leader_election.cc:288] T 00000000000000000000000000000000 P 1bb6de7c4f944007b1a42ba54b1b5581 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:47:12.426221 27575 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bb6de7c4f944007b1a42ba54b1b5581 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:47:12.426941 27575 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bb6de7c4f944007b1a42ba54b1b5581 [term 1 LEADER]: Becoming Leader. State: Replica: 1bb6de7c4f944007b1a42ba54b1b5581, State: Running, Role: LEADER
I20260620 18:47:12.427502 27575 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bb6de7c4f944007b1a42ba54b1b5581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb6de7c4f944007b1a42ba54b1b5581" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38855 } }
I20260620 18:47:12.428129 27565 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bb6de7c4f944007b1a42ba54b1b5581 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:47:12.430439 27576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bb6de7c4f944007b1a42ba54b1b5581 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bb6de7c4f944007b1a42ba54b1b5581" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb6de7c4f944007b1a42ba54b1b5581" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38855 } } }
I20260620 18:47:12.430799 27576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bb6de7c4f944007b1a42ba54b1b5581 [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:12.431864 27577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bb6de7c4f944007b1a42ba54b1b5581 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bb6de7c4f944007b1a42ba54b1b5581. Latest consensus state: current_term: 1 leader_uuid: "1bb6de7c4f944007b1a42ba54b1b5581" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb6de7c4f944007b1a42ba54b1b5581" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38855 } } }
I20260620 18:47:12.432281 27577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bb6de7c4f944007b1a42ba54b1b5581 [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:12.433835 27582 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:47:12.439116 27582 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:47:12.448889 27582 catalog_manager.cc:1347] Generated new cluster ID: c37d9488fd4d461cb4be00b382dd7af4
I20260620 18:47:12.449041 27582 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:47:12.475095 27582 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:47:12.476502 27582 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:47:12.485925 27582 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bb6de7c4f944007b1a42ba54b1b5581: Generated new TSK 0
I20260620 18:47:12.486852 27582 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:47:12.495092 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 27498
W20260620 18:47:12.496755 27560 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.164:57479
W20260620 18:47:12.497553 6022 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 27498: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:47:12.497712 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 27498
W20260620 18:47:12.514854 27483 connection.cc:535] client connection to 127.0.0.1:38855 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38855): Connection reset by peer (error 104)
2026-06-20T18:47:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1915 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:47:12 dist-test-slave-k6lg krb5kdc[27596](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:47:12 dist-test-slave-k6lg krb5kdc[27596](info): set up 2 sockets
Jun 20 18:47:12 dist-test-slave-k6lg krb5kdc[27596](info): commencing operation
krb5kdc: starting...
W20260620 18:47:13.610831 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:47:13 dist-test-slave-k6lg krb5kdc[27596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:47:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:47:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:47:13.862449 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33603
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33603
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:47:14.189332 27609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:14.189616 27609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:14.189781 27609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:14.196873 27609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:47:14.196992 27609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:14.197085 27609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:14.197172 27609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:47:14.197260 27609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:47:14.207188 27609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33603
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980894474392-6022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:47:14.208488 27609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:14.209867 27609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:14.222441 27615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:14.222687 27614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:14.222353 27617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:14.223687 27609 server_base.cc:1034] running on GCE node
I20260620 18:47:14.224546 27609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:14.225994 27609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:14.227240 27609 hybrid_clock.cc:648] HybridClock initialized: now 1781981234227202 us; error 56 us; skew 500 ppm
I20260620 18:47:14.227602 27609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:14 dist-test-slave-k6lg krb5kdc[27596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981234, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:14.230867 27609 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:47:14.232235 27609 webserver.cc:460] Webserver started at http://127.0.0.1:39155/ using document root <none> and password file <none>
I20260620 18:47:14.232844 27609 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:14.232944 27609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:14.233189 27609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:47:14.236152 27609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "9449f59dc27a4ad59b47c68cd46584a5"
format_stamp: "Formatted at 2026-06-20 18:47:14 on dist-test-slave-k6lg"
server_key: "4bc7599f45478c18877159ade566734c"
server_key_iv: "92ad7ed7238ac049a202c987a8f95f0b"
server_key_version: "encryptionkey@0"
I20260620 18:47:14.236856 27609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "9449f59dc27a4ad59b47c68cd46584a5"
format_stamp: "Formatted at 2026-06-20 18:47:14 on dist-test-slave-k6lg"
server_key: "4bc7599f45478c18877159ade566734c"
server_key_iv: "92ad7ed7238ac049a202c987a8f95f0b"
server_key_version: "encryptionkey@0"
I20260620 18:47:14.241822 27609 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:47:14.245141 27624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:14.246481 27609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 18:47:14.246649 27609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "9449f59dc27a4ad59b47c68cd46584a5"
format_stamp: "Formatted at 2026-06-20 18:47:14 on dist-test-slave-k6lg"
server_key: "4bc7599f45478c18877159ade566734c"
server_key_iv: "92ad7ed7238ac049a202c987a8f95f0b"
server_key_version: "encryptionkey@0"
I20260620 18:47:14.246811 27609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:14.252310 27609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:14.256517 27609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:14.256793 27609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:14.275184 27609 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33603
I20260620 18:47:14.275243 27675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33603 every 8 connection(s)
I20260620 18:47:14.276618 27609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:47:14.276995 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 27609
I20260620 18:47:14.277194 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:47:14.277799 6022 external_mini_cluster.cc:1428] Setting key 61ed73b56f6da632ad5b7387cf4c5966
I20260620 18:47:14.281170 27676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:47:14.285710 27679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:14.279728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:38191 (local address 127.0.0.1:33603)
0620 18:47:14.280463 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:14.280475 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:14.280517 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:47:14.282282 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:14.282305 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:14.282638 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:14.283016 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:14.284083 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:14.284089 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:14.284102 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:47:14.284117 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:14.284198 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:47:14.284217 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:14.284445 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:14.284633 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 18:47:14.284802 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":310,"threads_started":1}
I20260620 18:47:14.289269 6022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:47:14.292294 27676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9449f59dc27a4ad59b47c68cd46584a5: Bootstrap starting.
I20260620 18:47:14.296047 27676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9449f59dc27a4ad59b47c68cd46584a5: Neither blocks nor log segments found. Creating new log.
I20260620 18:47:14.296075 27679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:14.293669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:49807 (local address 127.0.0.1:33603)
0620 18:47:14.293876 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:14.293883 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:14.294224 (+ 341us) server_negotiation.cc:407] Connection header received
0620 18:47:14.294403 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:14.294406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:14.294452 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:14.294528 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:14.295443 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:14.295447 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:14.295449 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:47:14.295463 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:14.295497 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:47:14.295511 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:14.295595 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:14.295734 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 18:47:14.295778 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 18:47:14.297345 27676 log.cc:824] T 00000000000000000000000000000000 P 9449f59dc27a4ad59b47c68cd46584a5: Log is configured to *not* fsync() on all Append() calls
I20260620 18:47:14.299914 27676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9449f59dc27a4ad59b47c68cd46584a5: No bootstrap required, opened a new log
I20260620 18:47:14.307047 27676 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9449f59dc27a4ad59b47c68cd46584a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9449f59dc27a4ad59b47c68cd46584a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33603 } }
I20260620 18:47:14.307410 27676 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9449f59dc27a4ad59b47c68cd46584a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:47:14.307510 27676 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9449f59dc27a4ad59b47c68cd46584a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9449f59dc27a4ad59b47c68cd46584a5, State: Initialized, Role: FOLLOWER
I20260620 18:47:14.308034 27676 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9449f59dc27a4ad59b47c68cd46584a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9449f59dc27a4ad59b47c68cd46584a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33603 } }
I20260620 18:47:14.308280 27676 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9449f59dc27a4ad59b47c68cd46584a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:47:14.308429 27676 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9449f59dc27a4ad59b47c68cd46584a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:47:14.308597 27676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9449f59dc27a4ad59b47c68cd46584a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:47:14.310808 27676 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9449f59dc27a4ad59b47c68cd46584a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9449f59dc27a4ad59b47c68cd46584a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33603 } }
I20260620 18:47:14.311275 27676 leader_election.cc:302] T 00000000000000000000000000000000 P 9449f59dc27a4ad59b47c68cd46584a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9449f59dc27a4ad59b47c68cd46584a5; no voters:
I20260620 18:47:14.311825 27676 leader_election.cc:288] T 00000000000000000000000000000000 P 9449f59dc27a4ad59b47c68cd46584a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:47:14.311956 27686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9449f59dc27a4ad59b47c68cd46584a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:47:14.312611 27686 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9449f59dc27a4ad59b47c68cd46584a5 [term 1 LEADER]: Becoming Leader. State: Replica: 9449f59dc27a4ad59b47c68cd46584a5, State: Running, Role: LEADER
I20260620 18:47:14.313133 27686 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9449f59dc27a4ad59b47c68cd46584a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9449f59dc27a4ad59b47c68cd46584a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33603 } }
I20260620 18:47:14.313781 27676 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9449f59dc27a4ad59b47c68cd46584a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:47:14.316339 27687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9449f59dc27a4ad59b47c68cd46584a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9449f59dc27a4ad59b47c68cd46584a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9449f59dc27a4ad59b47c68cd46584a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33603 } } }
I20260620 18:47:14.316418 27688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9449f59dc27a4ad59b47c68cd46584a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9449f59dc27a4ad59b47c68cd46584a5. Latest consensus state: current_term: 1 leader_uuid: "9449f59dc27a4ad59b47c68cd46584a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9449f59dc27a4ad59b47c68cd46584a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33603 } } }
I20260620 18:47:14.316848 27688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9449f59dc27a4ad59b47c68cd46584a5 [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:14.316838 27687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9449f59dc27a4ad59b47c68cd46584a5 [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:14.317814 27693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:47:14.325004 27693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:47:14.334748 27693 catalog_manager.cc:1347] Generated new cluster ID: 8b88eaa7f16f4839bc80bfe3b2f85f40
I20260620 18:47:14.334878 27693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:47:14.354722 27693 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:47:14.355254 27693 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:47:14.366335 27693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9449f59dc27a4ad59b47c68cd46584a5: Generated new TSK 0
I20260620 18:47:14.367125 27693 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:47:14.378937 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 27609
W20260620 18:47:14.380282 27671 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.164:38191
W20260620 18:47:14.380932 6022 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 27609: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:47:14.381106 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 27609
2026-06-20T18:47:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1878 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:47:14 dist-test-slave-k6lg krb5kdc[27707](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:47:14 dist-test-slave-k6lg krb5kdc[27707](info): set up 2 sockets
Jun 20 18:47:14 dist-test-slave-k6lg krb5kdc[27707](info): commencing operation
krb5kdc: starting...
W20260620 18:47:15.494768 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:47:15 dist-test-slave-k6lg krb5kdc[27707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:47:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:47:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
I20260620 18:47:15.751353 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:36423
--webserver_interface=127.5.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:35679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:36423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:47:16.075616 27720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:16.075897 27720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:16.076066 27720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:16.083300 27720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:47:16.083413 27720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:16.083535 27720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:16.083633 27720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:47:16.083693 27720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:47:16.093364 27720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:35679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:36423
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:36423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:47:16.094695 27720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:16.095897 27720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:16.107144 27726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:16.107247 27725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:16.108541 27728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:16.108732 27720 server_base.cc:1034] running on GCE node
I20260620 18:47:16.109715 27720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:16.111408 27720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:16.112766 27720 hybrid_clock.cc:648] HybridClock initialized: now 1781981236112704 us; error 43 us; skew 500 ppm
I20260620 18:47:16.113201 27720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:16 dist-test-slave-k6lg krb5kdc[27707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981236, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:16.116559 27720 init.cc:377] Logged in from keytab as kudu/127.5.225.190@KRBTEST.COM (short username kudu)
I20260620 18:47:16.118245 27720 webserver.cc:460] Webserver started at http://127.5.225.190:37851/ using document root <none> and password file <none>
I20260620 18:47:16.118978 27720 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:16.119136 27720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:16.119460 27720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:47:16.122648 27720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "75b29cb19b4e4c5d9ff8e06f228c8df0"
format_stamp: "Formatted at 2026-06-20 18:47:16 on dist-test-slave-k6lg"
server_key: "301e4d0353f36cc9bc80ba5790945268"
server_key_iv: "1252414289f3e502a0b50fa160c4861a"
server_key_version: "encryptionkey@0"
I20260620 18:47:16.123438 27720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "75b29cb19b4e4c5d9ff8e06f228c8df0"
format_stamp: "Formatted at 2026-06-20 18:47:16 on dist-test-slave-k6lg"
server_key: "301e4d0353f36cc9bc80ba5790945268"
server_key_iv: "1252414289f3e502a0b50fa160c4861a"
server_key_version: "encryptionkey@0"
I20260620 18:47:16.128755 27720 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:47:16.132648 27735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:16.134006 27720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 18:47:16.134265 27720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "75b29cb19b4e4c5d9ff8e06f228c8df0"
format_stamp: "Formatted at 2026-06-20 18:47:16 on dist-test-slave-k6lg"
server_key: "301e4d0353f36cc9bc80ba5790945268"
server_key_iv: "1252414289f3e502a0b50fa160c4861a"
server_key_version: "encryptionkey@0"
I20260620 18:47:16.134472 27720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:16.155988 27720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:16.160039 27720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:16.160331 27720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:16.179558 27720 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:36423
I20260620 18:47:16.179570 27786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:36423 every 8 connection(s)
I20260620 18:47:16.180933 27720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:47:16.184671 27787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:47:16.185497 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 27720
I20260620 18:47:16.185681 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:47:16.186146 6022 external_mini_cluster.cc:1428] Setting key 1a34672979d946e396aa907dbabe7842
I20260620 18:47:16.199211 27787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75b29cb19b4e4c5d9ff8e06f228c8df0: Bootstrap starting.
Jun 20 18:47:16 dist-test-slave-k6lg krb5kdc[27707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:47:16.202330 27787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75b29cb19b4e4c5d9ff8e06f228c8df0: Neither blocks nor log segments found. Creating new log.
I20260620 18:47:16.203755 27787 log.cc:824] T 00000000000000000000000000000000 P 75b29cb19b4e4c5d9ff8e06f228c8df0: Log is configured to *not* fsync() on all Append() calls
I20260620 18:47:16.207175 27787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75b29cb19b4e4c5d9ff8e06f228c8df0: No bootstrap required, opened a new log
I20260620 18:47:16.211781 27790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:16.188137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40575 (local address 127.5.225.190:36423)
0620 18:47:16.188946 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:16.188960 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:16.189009 (+ 49us) server_negotiation.cc:407] Connection header received
0620 18:47:16.191489 (+ 2480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:16.191519 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:16.192061 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:16.192543 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:16.193940 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:16.195987 (+ 2047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:16.197245 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:16.198341 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:16.198529 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:16.202685 (+ 4156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:16.202702 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:16.202723 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:16.202776 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:16.206860 (+ 4084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:16.207773 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:16.207781 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:16.207788 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:16.208000 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:16.208436 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:16.208441 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:16.208444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:16.209147 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:16.209488 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:16.209978 (+ 490us) server_negotiation.cc:299] Negotiation successful
0620 18:47:16.210363 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":357,"threads_started":1}
I20260620 18:47:16.215857 6022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:47:16.217016 27787 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75b29cb19b4e4c5d9ff8e06f228c8df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75b29cb19b4e4c5d9ff8e06f228c8df0" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36423 } }
I20260620 18:47:16.217386 27787 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75b29cb19b4e4c5d9ff8e06f228c8df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:47:16.217504 27787 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75b29cb19b4e4c5d9ff8e06f228c8df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75b29cb19b4e4c5d9ff8e06f228c8df0, State: Initialized, Role: FOLLOWER
I20260620 18:47:16.218107 27787 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75b29cb19b4e4c5d9ff8e06f228c8df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75b29cb19b4e4c5d9ff8e06f228c8df0" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36423 } }
I20260620 18:47:16.218430 27787 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75b29cb19b4e4c5d9ff8e06f228c8df0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:47:16.218588 27787 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75b29cb19b4e4c5d9ff8e06f228c8df0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:47:16.218801 27787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75b29cb19b4e4c5d9ff8e06f228c8df0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:47:16.221776 27787 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75b29cb19b4e4c5d9ff8e06f228c8df0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75b29cb19b4e4c5d9ff8e06f228c8df0" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36423 } }
I20260620 18:47:16.222429 27787 leader_election.cc:302] T 00000000000000000000000000000000 P 75b29cb19b4e4c5d9ff8e06f228c8df0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75b29cb19b4e4c5d9ff8e06f228c8df0; no voters:
I20260620 18:47:16.223136 27787 leader_election.cc:288] T 00000000000000000000000000000000 P 75b29cb19b4e4c5d9ff8e06f228c8df0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:47:16.223263 27797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75b29cb19b4e4c5d9ff8e06f228c8df0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:47:16.225315 27787 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75b29cb19b4e4c5d9ff8e06f228c8df0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:47:16.225452 27797 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75b29cb19b4e4c5d9ff8e06f228c8df0 [term 1 LEADER]: Becoming Leader. State: Replica: 75b29cb19b4e4c5d9ff8e06f228c8df0, State: Running, Role: LEADER
I20260620 18:47:16.226004 27797 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75b29cb19b4e4c5d9ff8e06f228c8df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75b29cb19b4e4c5d9ff8e06f228c8df0" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36423 } }
I20260620 18:47:16.229082 27798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75b29cb19b4e4c5d9ff8e06f228c8df0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75b29cb19b4e4c5d9ff8e06f228c8df0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75b29cb19b4e4c5d9ff8e06f228c8df0" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36423 } } }
I20260620 18:47:16.229508 27798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75b29cb19b4e4c5d9ff8e06f228c8df0 [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:16.229979 27799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75b29cb19b4e4c5d9ff8e06f228c8df0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75b29cb19b4e4c5d9ff8e06f228c8df0. Latest consensus state: current_term: 1 leader_uuid: "75b29cb19b4e4c5d9ff8e06f228c8df0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75b29cb19b4e4c5d9ff8e06f228c8df0" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 36423 } } }
I20260620 18:47:16.230304 27799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75b29cb19b4e4c5d9ff8e06f228c8df0 [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:16.239185 27803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:47:16.241089 27790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:16.220575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37869 (local address 127.5.225.190:36423)
0620 18:47:16.220783 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:16.220788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:16.221081 (+ 293us) server_negotiation.cc:407] Connection header received
0620 18:47:16.221256 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:16.221259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:16.221323 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:16.221398 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:16.223388 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:16.225013 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:16.226972 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:16.227635 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:16.227796 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:16.232816 (+ 5020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:16.232826 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:16.232834 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:16.232888 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:16.237630 (+ 4742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:16.238870 (+ 1240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:16.238876 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:16.238879 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:16.239018 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:16.239521 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:16.239526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:16.239529 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:16.239844 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:16.240124 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:16.240529 (+ 405us) server_negotiation.cc:299] Negotiation successful
0620 18:47:16.240713 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 18:47:16.250703 27803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:47:16.259678 27803 catalog_manager.cc:1347] Generated new cluster ID: eb74422e612e4e279412be887f648763
I20260620 18:47:16.259784 27803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:47:16.274978 27803 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:47:16.276346 27803 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:47:16.290241 27803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75b29cb19b4e4c5d9ff8e06f228c8df0: Generated new TSK 0
I20260620 18:47:16.291096 27803 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:47:16.298491 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 27720
I20260620 18:47:16.300251 27782 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:47:16.392947 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 27720
W20260620 18:47:16.417486 27705 connection.cc:535] client connection to 127.5.225.190:36423 recv error: Network error: failed to read from TLS socket (remote: 127.5.225.190:36423): Connection reset by peer (error 104)
2026-06-20T18:47:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2019 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:47:16 dist-test-slave-k6lg krb5kdc[27819](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:47:16 dist-test-slave-k6lg krb5kdc[27819](info): set up 2 sockets
Jun 20 18:47:16 dist-test-slave-k6lg krb5kdc[27819](info): commencing operation
krb5kdc: starting...
W20260620 18:47:17.510694 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:47:17 dist-test-slave-k6lg krb5kdc[27819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:47:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:47:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
I20260620 18:47:17.755460 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:39977
--webserver_interface=127.5.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:34595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:39977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:47:18.080216 27832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:18.080579 27832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:18.080754 27832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:18.087942 27832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:47:18.088171 27832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:18.088279 27832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:18.088390 27832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:47:18.088476 27832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:47:18.098301 27832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:34595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:39977
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:39977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:47:18.099675 27832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:18.101045 27832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:18.112876 27837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:18.112843 27838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:18.112959 27840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:18.113889 27832 server_base.cc:1034] running on GCE node
I20260620 18:47:18.114759 27832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:18.116164 27832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:18.117398 27832 hybrid_clock.cc:648] HybridClock initialized: now 1781981238117352 us; error 41 us; skew 500 ppm
I20260620 18:47:18.117784 27832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:18 dist-test-slave-k6lg krb5kdc[27819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981238, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:18.121011 27832 init.cc:377] Logged in from keytab as kudu/127.5.225.190@KRBTEST.COM (short username kudu)
I20260620 18:47:18.122555 27832 webserver.cc:460] Webserver started at http://127.5.225.190:46199/ using document root <none> and password file <none>
I20260620 18:47:18.123250 27832 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:18.123397 27832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:18.123713 27832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:47:18.126972 27832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "1cb4228365264d189a93855b577441cc"
format_stamp: "Formatted at 2026-06-20 18:47:18 on dist-test-slave-k6lg"
server_key: "dafa59133a63f2eb6b749624fdab0b7b"
server_key_iv: "d0bc6ef4a00ddba0827318baf69d7171"
server_key_version: "encryptionkey@0"
I20260620 18:47:18.127856 27832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "1cb4228365264d189a93855b577441cc"
format_stamp: "Formatted at 2026-06-20 18:47:18 on dist-test-slave-k6lg"
server_key: "dafa59133a63f2eb6b749624fdab0b7b"
server_key_iv: "d0bc6ef4a00ddba0827318baf69d7171"
server_key_version: "encryptionkey@0"
I20260620 18:47:18.133196 27832 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260620 18:47:18.136814 27847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:18.138151 27832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 18:47:18.138409 27832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "1cb4228365264d189a93855b577441cc"
format_stamp: "Formatted at 2026-06-20 18:47:18 on dist-test-slave-k6lg"
server_key: "dafa59133a63f2eb6b749624fdab0b7b"
server_key_iv: "d0bc6ef4a00ddba0827318baf69d7171"
server_key_version: "encryptionkey@0"
I20260620 18:47:18.138609 27832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:18.163232 27832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:18.167945 27832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:18.168262 27832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:18.187747 27832 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:39977
I20260620 18:47:18.187769 27898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:39977 every 8 connection(s)
I20260620 18:47:18.189263 27832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:47:18.193312 27899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:47:18.199292 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 27832
I20260620 18:47:18.199487 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:47:18.200042 6022 external_mini_cluster.cc:1428] Setting key f0d073391049d8c1415ebc0ed7812151
I20260620 18:47:18.204546 27899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cb4228365264d189a93855b577441cc: Bootstrap starting.
I20260620 18:47:18.208252 27899 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cb4228365264d189a93855b577441cc: Neither blocks nor log segments found. Creating new log.
I20260620 18:47:18.209539 27899 log.cc:824] T 00000000000000000000000000000000 P 1cb4228365264d189a93855b577441cc: Log is configured to *not* fsync() on all Append() calls
I20260620 18:47:18.213110 27899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cb4228365264d189a93855b577441cc: No bootstrap required, opened a new log
I20260620 18:47:18.214618 27902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:18.201922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57339 (local address 127.5.225.190:39977)
0620 18:47:18.202877 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:18.202889 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:18.202930 (+ 41us) server_negotiation.cc:407] Connection header received
0620 18:47:18.204576 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:18.204599 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:18.205039 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:18.205361 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:18.205930 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:18.207328 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:18.209276 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:18.210205 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:18.210426 (+ 221us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:18.211617 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:18.211627 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:18.211646 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:47:18.212431 (+ 785us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:18.212533 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:47:18.212565 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:18.212719 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:18.213047 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 18:47:18.213234 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":733,"thread_start_us":472,"threads_started":1}
I20260620 18:47:18.219256 6022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:47:18.222425 27899 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cb4228365264d189a93855b577441cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb4228365264d189a93855b577441cc" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39977 } }
I20260620 18:47:18.222836 27899 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cb4228365264d189a93855b577441cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:47:18.222944 27899 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cb4228365264d189a93855b577441cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb4228365264d189a93855b577441cc, State: Initialized, Role: FOLLOWER
I20260620 18:47:18.223603 27899 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cb4228365264d189a93855b577441cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb4228365264d189a93855b577441cc" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39977 } }
I20260620 18:47:18.223985 27899 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cb4228365264d189a93855b577441cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:47:18.224136 27899 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cb4228365264d189a93855b577441cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:47:18.224342 27899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cb4228365264d189a93855b577441cc [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:47:18.227393 27899 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cb4228365264d189a93855b577441cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb4228365264d189a93855b577441cc" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39977 } }
I20260620 18:47:18.228030 27899 leader_election.cc:302] T 00000000000000000000000000000000 P 1cb4228365264d189a93855b577441cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cb4228365264d189a93855b577441cc; no voters:
I20260620 18:47:18.228761 27899 leader_election.cc:288] T 00000000000000000000000000000000 P 1cb4228365264d189a93855b577441cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:47:18.228891 27909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cb4228365264d189a93855b577441cc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:47:18.230192 27909 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cb4228365264d189a93855b577441cc [term 1 LEADER]: Becoming Leader. State: Replica: 1cb4228365264d189a93855b577441cc, State: Running, Role: LEADER
I20260620 18:47:18.231133 27902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:18.223770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48687 (local address 127.5.225.190:39977)
0620 18:47:18.223964 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:18.223975 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:18.224305 (+ 330us) server_negotiation.cc:407] Connection header received
0620 18:47:18.224561 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:18.224566 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:18.224655 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:18.224758 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:18.225231 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:18.226439 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:18.227786 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:18.228459 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:18.228641 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:18.229875 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:18.229885 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:18.229889 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:47:18.229918 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:18.230003 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:47:18.230024 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:18.230557 (+ 533us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:18.230651 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 18:47:18.230709 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:47:18.230952 27909 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cb4228365264d189a93855b577441cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb4228365264d189a93855b577441cc" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39977 } }
I20260620 18:47:18.231374 27899 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cb4228365264d189a93855b577441cc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:47:18.235241 27911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cb4228365264d189a93855b577441cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cb4228365264d189a93855b577441cc. Latest consensus state: current_term: 1 leader_uuid: "1cb4228365264d189a93855b577441cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb4228365264d189a93855b577441cc" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39977 } } }
I20260620 18:47:18.235219 27910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cb4228365264d189a93855b577441cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cb4228365264d189a93855b577441cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb4228365264d189a93855b577441cc" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39977 } } }
I20260620 18:47:18.235749 27911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cb4228365264d189a93855b577441cc [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:18.235759 27910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cb4228365264d189a93855b577441cc [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:18.236761 27917 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:47:18.243481 27917 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:47:18.254253 27917 catalog_manager.cc:1347] Generated new cluster ID: 8b47e904b2224c26b16979794dee52fa
I20260620 18:47:18.254442 27917 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:47:18.274569 27917 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:47:18.275892 27917 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:47:18.286767 27917 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cb4228365264d189a93855b577441cc: Generated new TSK 0
I20260620 18:47:18.287585 27917 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:47:18.298228 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 27832
W20260620 18:47:18.299285 27894 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57339
W20260620 18:47:18.300031 6022 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 27832: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:47:18.300264 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 27832
W20260620 18:47:18.317174 27817 connection.cc:535] client connection to 127.5.225.190:39977 recv error: Network error: failed to read from TLS socket (remote: 127.5.225.190:39977): Connection reset by peer (error 104)
2026-06-20T18:47:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1899 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:47:18 dist-test-slave-k6lg krb5kdc[27930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:47:18 dist-test-slave-k6lg krb5kdc[27930](info): set up 2 sockets
Jun 20 18:47:18 dist-test-slave-k6lg krb5kdc[27930](info): commencing operation
krb5kdc: starting...
W20260620 18:47:19.410616 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.008s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:47:19 dist-test-slave-k6lg krb5kdc[27930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:47:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:47:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
I20260620 18:47:19.662021 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:44895
--webserver_interface=127.5.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:33595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:44895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:47:19.984704 27943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:19.985035 27943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:19.985206 27943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:19.992499 27943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:47:19.992664 27943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:19.992781 27943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:19.992901 27943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:47:19.992995 27943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:47:20.003090 27943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:33595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:44895
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:44895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:47:20.004495 27943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:20.005890 27943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:20.018153 27949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:20.018101 27951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:20.018198 27948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:20.018904 27943 server_base.cc:1034] running on GCE node
I20260620 18:47:20.019737 27943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:20.021198 27943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:20.022397 27943 hybrid_clock.cc:648] HybridClock initialized: now 1781981240022349 us; error 43 us; skew 500 ppm
I20260620 18:47:20.022755 27943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:20 dist-test-slave-k6lg krb5kdc[27930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981240, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:20.026093 27943 init.cc:377] Logged in from keytab as kudu/127.5.225.190@KRBTEST.COM (short username kudu)
I20260620 18:47:20.027792 27943 webserver.cc:460] Webserver started at http://127.5.225.190:34135/ using document root <none> and password file <none>
I20260620 18:47:20.028508 27943 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:20.028651 27943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:20.029022 27943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:47:20.032294 27943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "5fa96df35b2646668a9f54a834b525e8"
format_stamp: "Formatted at 2026-06-20 18:47:20 on dist-test-slave-k6lg"
server_key: "eb963e62d21243713e900242094336ea"
server_key_iv: "6a4b270b36d71325cd83433fd1c16807"
server_key_version: "encryptionkey@0"
I20260620 18:47:20.033129 27943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "5fa96df35b2646668a9f54a834b525e8"
format_stamp: "Formatted at 2026-06-20 18:47:20 on dist-test-slave-k6lg"
server_key: "eb963e62d21243713e900242094336ea"
server_key_iv: "6a4b270b36d71325cd83433fd1c16807"
server_key_version: "encryptionkey@0"
I20260620 18:47:20.038614 27943 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:47:20.042385 27958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:20.043838 27943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 18:47:20.044044 27943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "5fa96df35b2646668a9f54a834b525e8"
format_stamp: "Formatted at 2026-06-20 18:47:20 on dist-test-slave-k6lg"
server_key: "eb963e62d21243713e900242094336ea"
server_key_iv: "6a4b270b36d71325cd83433fd1c16807"
server_key_version: "encryptionkey@0"
I20260620 18:47:20.044222 27943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:20.050026 27943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:20.053965 27943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:20.054324 27943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:20.072564 27943 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:44895
I20260620 18:47:20.072595 28009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:44895 every 8 connection(s)
I20260620 18:47:20.073925 27943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:47:20.075099 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 27943
I20260620 18:47:20.075300 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:47:20.075843 6022 external_mini_cluster.cc:1428] Setting key c1bc1448f838695b14ba286823691cc0
I20260620 18:47:20.078526 28010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:47:20.083665 28013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:20.077734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47479 (local address 127.5.225.190:44895)
0620 18:47:20.078556 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:20.078589 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:20.078642 (+ 53us) server_negotiation.cc:407] Connection header received
0620 18:47:20.080269 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:20.080289 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:20.080614 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:20.080947 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:20.082054 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:20.082060 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:20.082072 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:47:20.082089 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:20.082218 (+ 129us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:47:20.082249 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:20.082429 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:20.082543 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 18:47:20.082771 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":383,"threads_started":1}
I20260620 18:47:20.087221 6022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:47:20.090756 28010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fa96df35b2646668a9f54a834b525e8: Bootstrap starting.
I20260620 18:47:20.094787 28010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fa96df35b2646668a9f54a834b525e8: Neither blocks nor log segments found. Creating new log.
I20260620 18:47:20.095108 28013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:20.092467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43631 (local address 127.5.225.190:44895)
0620 18:47:20.092675 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:20.092681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:20.092996 (+ 315us) server_negotiation.cc:407] Connection header received
0620 18:47:20.093214 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:20.093218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:20.093265 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:20.093350 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:20.094366 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:20.094370 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:20.094373 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:47:20.094388 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:20.094424 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:47:20.094439 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:20.094538 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:20.094700 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 18:47:20.094743 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 18:47:20.096005 28010 log.cc:824] T 00000000000000000000000000000000 P 5fa96df35b2646668a9f54a834b525e8: Log is configured to *not* fsync() on all Append() calls
I20260620 18:47:20.098963 28010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fa96df35b2646668a9f54a834b525e8: No bootstrap required, opened a new log
I20260620 18:47:20.106341 28010 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fa96df35b2646668a9f54a834b525e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa96df35b2646668a9f54a834b525e8" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 44895 } }
I20260620 18:47:20.106750 28010 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fa96df35b2646668a9f54a834b525e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:47:20.106861 28010 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fa96df35b2646668a9f54a834b525e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fa96df35b2646668a9f54a834b525e8, State: Initialized, Role: FOLLOWER
I20260620 18:47:20.107501 28010 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fa96df35b2646668a9f54a834b525e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa96df35b2646668a9f54a834b525e8" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 44895 } }
I20260620 18:47:20.107699 28010 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fa96df35b2646668a9f54a834b525e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:47:20.107798 28010 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fa96df35b2646668a9f54a834b525e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:47:20.107933 28010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fa96df35b2646668a9f54a834b525e8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:47:20.110081 28010 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fa96df35b2646668a9f54a834b525e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa96df35b2646668a9f54a834b525e8" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 44895 } }
I20260620 18:47:20.110682 28010 leader_election.cc:302] T 00000000000000000000000000000000 P 5fa96df35b2646668a9f54a834b525e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fa96df35b2646668a9f54a834b525e8; no voters:
I20260620 18:47:20.111374 28010 leader_election.cc:288] T 00000000000000000000000000000000 P 5fa96df35b2646668a9f54a834b525e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:47:20.111500 28020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fa96df35b2646668a9f54a834b525e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:47:20.112170 28020 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fa96df35b2646668a9f54a834b525e8 [term 1 LEADER]: Becoming Leader. State: Replica: 5fa96df35b2646668a9f54a834b525e8, State: Running, Role: LEADER
I20260620 18:47:20.112783 28020 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fa96df35b2646668a9f54a834b525e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa96df35b2646668a9f54a834b525e8" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 44895 } }
I20260620 18:47:20.113631 28010 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fa96df35b2646668a9f54a834b525e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:47:20.115694 28022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fa96df35b2646668a9f54a834b525e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fa96df35b2646668a9f54a834b525e8. Latest consensus state: current_term: 1 leader_uuid: "5fa96df35b2646668a9f54a834b525e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa96df35b2646668a9f54a834b525e8" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 44895 } } }
I20260620 18:47:20.116117 28022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fa96df35b2646668a9f54a834b525e8 [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:20.116490 28021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fa96df35b2646668a9f54a834b525e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fa96df35b2646668a9f54a834b525e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa96df35b2646668a9f54a834b525e8" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 44895 } } }
I20260620 18:47:20.116820 28021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fa96df35b2646668a9f54a834b525e8 [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:20.118834 28029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:47:20.124164 28029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:47:20.132427 28029 catalog_manager.cc:1347] Generated new cluster ID: 15616a72e5fe47b4bb21ac79368df301
I20260620 18:47:20.132550 28029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:47:20.158316 28029 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:47:20.158754 28029 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:47:20.164237 28029 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fa96df35b2646668a9f54a834b525e8: Generated new TSK 0
I20260620 18:47:20.165030 28029 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:47:20.170930 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 27943
W20260620 18:47:20.172024 28005 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47479
W20260620 18:47:20.172703 6022 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 27943: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:47:20.172843 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 27943
2026-06-20T18:47:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1872 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:47:20 dist-test-slave-k6lg krb5kdc[28041](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:47:20 dist-test-slave-k6lg krb5kdc[28041](info): set up 2 sockets
Jun 20 18:47:20 dist-test-slave-k6lg krb5kdc[28041](info): commencing operation
krb5kdc: starting...
W20260620 18:47:21.270810 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.003s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:47:21 dist-test-slave-k6lg krb5kdc[28041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:47:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:47:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
I20260620 18:47:21.537859 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:42789
--webserver_interface=127.5.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:41857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:42789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:47:21.876695 28054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:21.876979 28054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:21.877074 28054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:21.884323 28054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:47:21.884434 28054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:21.884483 28054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:21.884536 28054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:47:21.884581 28054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:47:21.894387 28054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:41857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:42789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:42789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:47:21.895618 28054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:21.896890 28054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:21.908370 28059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:21.908633 28060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:21.908937 28062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:21.909636 28054 server_base.cc:1034] running on GCE node
I20260620 18:47:21.910557 28054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:21.912024 28054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:21.913246 28054 hybrid_clock.cc:648] HybridClock initialized: now 1781981241913204 us; error 32 us; skew 500 ppm
I20260620 18:47:21.913620 28054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:21 dist-test-slave-k6lg krb5kdc[28041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981241, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:21.920879 28054 init.cc:377] Logged in from keytab as kudu/127.5.225.190@KRBTEST.COM (short username kudu)
I20260620 18:47:21.922565 28054 webserver.cc:460] Webserver started at http://127.5.225.190:39577/ using document root <none> and password file <none>
I20260620 18:47:21.923266 28054 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:21.923398 28054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:21.923727 28054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:47:21.927264 28054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "c1bf5ce8abd2436bab1fd2e9bcded4af"
format_stamp: "Formatted at 2026-06-20 18:47:21 on dist-test-slave-k6lg"
server_key: "ede8175070f1763b57d9a2471872587c"
server_key_iv: "faf1193e5784f36232463234d8ef7428"
server_key_version: "encryptionkey@0"
I20260620 18:47:21.928046 28054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "c1bf5ce8abd2436bab1fd2e9bcded4af"
format_stamp: "Formatted at 2026-06-20 18:47:21 on dist-test-slave-k6lg"
server_key: "ede8175070f1763b57d9a2471872587c"
server_key_iv: "faf1193e5784f36232463234d8ef7428"
server_key_version: "encryptionkey@0"
I20260620 18:47:21.933282 28054 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:47:21.937163 28069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:21.938825 28054 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 18:47:21.939038 28054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "c1bf5ce8abd2436bab1fd2e9bcded4af"
format_stamp: "Formatted at 2026-06-20 18:47:21 on dist-test-slave-k6lg"
server_key: "ede8175070f1763b57d9a2471872587c"
server_key_iv: "faf1193e5784f36232463234d8ef7428"
server_key_version: "encryptionkey@0"
I20260620 18:47:21.939224 28054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:21.972723 28054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:21.977123 28054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:21.977411 28054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:21.996762 28054 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:42789
I20260620 18:47:21.996784 28120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:42789 every 8 connection(s)
I20260620 18:47:21.998205 28054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:47:22.002097 28121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:47:22.003315 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 28054
I20260620 18:47:22.003496 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:47:22.004086 6022 external_mini_cluster.cc:1428] Setting key c7c23d7a5adb5c117df3886d32587256
I20260620 18:47:22.016553 28121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1bf5ce8abd2436bab1fd2e9bcded4af: Bootstrap starting.
Jun 20 18:47:22 dist-test-slave-k6lg krb5kdc[28041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:47:22.020171 28121 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1bf5ce8abd2436bab1fd2e9bcded4af: Neither blocks nor log segments found. Creating new log.
I20260620 18:47:22.021337 28121 log.cc:824] T 00000000000000000000000000000000 P c1bf5ce8abd2436bab1fd2e9bcded4af: Log is configured to *not* fsync() on all Append() calls
I20260620 18:47:22.024235 28121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1bf5ce8abd2436bab1fd2e9bcded4af: No bootstrap required, opened a new log
I20260620 18:47:22.027392 28124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:22.005965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60335 (local address 127.5.225.190:42789)
0620 18:47:22.006803 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:22.006818 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:22.006860 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:47:22.008602 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:22.008627 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:22.009027 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:22.009368 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:22.011208 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:22.012492 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:22.014315 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:22.014951 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:22.015082 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:22.019754 (+ 4672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:22.019762 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:22.019777 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:22.019814 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:22.023062 (+ 3248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:22.023744 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:22.023751 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:22.023756 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:22.023919 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:22.024328 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:22.024350 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:22.024356 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:22.024894 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:22.025327 (+ 433us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:22.025808 (+ 481us) server_negotiation.cc:299] Negotiation successful
0620 18:47:22.026356 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":419,"threads_started":1}
I20260620 18:47:22.031188 6022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:47:22.032716 28121 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1bf5ce8abd2436bab1fd2e9bcded4af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1bf5ce8abd2436bab1fd2e9bcded4af" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 42789 } }
I20260620 18:47:22.033082 28121 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1bf5ce8abd2436bab1fd2e9bcded4af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:47:22.033195 28121 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1bf5ce8abd2436bab1fd2e9bcded4af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1bf5ce8abd2436bab1fd2e9bcded4af, State: Initialized, Role: FOLLOWER
I20260620 18:47:22.033810 28121 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1bf5ce8abd2436bab1fd2e9bcded4af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1bf5ce8abd2436bab1fd2e9bcded4af" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 42789 } }
I20260620 18:47:22.034087 28121 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1bf5ce8abd2436bab1fd2e9bcded4af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:47:22.034271 28121 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1bf5ce8abd2436bab1fd2e9bcded4af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:47:22.034453 28121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1bf5ce8abd2436bab1fd2e9bcded4af [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:47:22.037268 28121 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1bf5ce8abd2436bab1fd2e9bcded4af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1bf5ce8abd2436bab1fd2e9bcded4af" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 42789 } }
I20260620 18:47:22.037757 28121 leader_election.cc:302] T 00000000000000000000000000000000 P c1bf5ce8abd2436bab1fd2e9bcded4af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1bf5ce8abd2436bab1fd2e9bcded4af; no voters:
I20260620 18:47:22.038399 28121 leader_election.cc:288] T 00000000000000000000000000000000 P c1bf5ce8abd2436bab1fd2e9bcded4af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:47:22.038499 28131 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1bf5ce8abd2436bab1fd2e9bcded4af [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:47:22.040532 28121 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1bf5ce8abd2436bab1fd2e9bcded4af [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:47:22.042527 28131 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1bf5ce8abd2436bab1fd2e9bcded4af [term 1 LEADER]: Becoming Leader. State: Replica: c1bf5ce8abd2436bab1fd2e9bcded4af, State: Running, Role: LEADER
I20260620 18:47:22.043236 28131 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1bf5ce8abd2436bab1fd2e9bcded4af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1bf5ce8abd2436bab1fd2e9bcded4af" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 42789 } }
I20260620 18:47:22.046195 28133 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1bf5ce8abd2436bab1fd2e9bcded4af [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1bf5ce8abd2436bab1fd2e9bcded4af. Latest consensus state: current_term: 1 leader_uuid: "c1bf5ce8abd2436bab1fd2e9bcded4af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1bf5ce8abd2436bab1fd2e9bcded4af" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 42789 } } }
I20260620 18:47:22.046598 28133 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1bf5ce8abd2436bab1fd2e9bcded4af [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:22.049499 28132 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1bf5ce8abd2436bab1fd2e9bcded4af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1bf5ce8abd2436bab1fd2e9bcded4af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1bf5ce8abd2436bab1fd2e9bcded4af" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 42789 } } }
I20260620 18:47:22.050635 28132 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1bf5ce8abd2436bab1fd2e9bcded4af [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:22.052443 28137 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:47:22.054721 28124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:22.035705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58685 (local address 127.5.225.190:42789)
0620 18:47:22.035897 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:22.035904 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:22.036329 (+ 425us) server_negotiation.cc:407] Connection header received
0620 18:47:22.036467 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:22.036472 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:22.036550 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:22.036662 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:22.038230 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:22.039465 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:22.041265 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:22.042353 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:22.043274 (+ 921us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:22.045784 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:22.045794 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:22.045801 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:22.045850 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:22.051257 (+ 5407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:22.052145 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:22.052151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:22.052155 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:22.052317 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:22.052844 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:22.052851 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:22.052858 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:22.053359 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:22.053663 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:22.054014 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 18:47:22.054266 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:47:22.058410 28137 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:47:22.067785 28137 catalog_manager.cc:1347] Generated new cluster ID: d75aa575ccec4af6801847e54d565440
I20260620 18:47:22.067906 28137 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:47:22.090708 28137 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:47:22.092675 28137 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:47:22.106084 28137 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1bf5ce8abd2436bab1fd2e9bcded4af: Generated new TSK 0
I20260620 18:47:22.107012 28137 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:47:22.114786 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 28054
I20260620 18:47:22.116559 28116 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:47:22.208362 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 28054
2026-06-20T18:47:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2035 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:47:22 dist-test-slave-k6lg krb5kdc[28153](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:47:22 dist-test-slave-k6lg krb5kdc[28153](info): set up 2 sockets
Jun 20 18:47:22 dist-test-slave-k6lg krb5kdc[28153](info): commencing operation
krb5kdc: starting...
W20260620 18:47:23.335318 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:47:23 dist-test-slave-k6lg krb5kdc[28153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:47:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:47:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
I20260620 18:47:23.585667 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:37665
--webserver_interface=127.5.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:36891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:37665
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:47:23.912314 28166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:23.912588 28166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:23.912744 28166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:23.919834 28166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:47:23.919938 28166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:23.920034 28166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:23.920135 28166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:47:23.920212 28166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:47:23.929778 28166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:36891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:37665
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:37665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:47:23.931118 28166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:23.932400 28166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:23.944254 28171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:23.944245 28174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:23.944295 28172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:23.945101 28166 server_base.cc:1034] running on GCE node
I20260620 18:47:23.945928 28166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:23.947419 28166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:23.948660 28166 hybrid_clock.cc:648] HybridClock initialized: now 1781981243948611 us; error 54 us; skew 500 ppm
I20260620 18:47:23.949018 28166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:23 dist-test-slave-k6lg krb5kdc[28153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981243, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:23.952313 28166 init.cc:377] Logged in from keytab as kudu/127.5.225.190@KRBTEST.COM (short username kudu)
I20260620 18:47:23.953859 28166 webserver.cc:460] Webserver started at http://127.5.225.190:38343/ using document root <none> and password file <none>
I20260620 18:47:23.954609 28166 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:23.954756 28166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:23.955061 28166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:47:23.957993 28166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "49ccd525618543ad97e0e5a581631fae"
format_stamp: "Formatted at 2026-06-20 18:47:23 on dist-test-slave-k6lg"
server_key: "180c1f90710aa8bdecec29d3cceaec4e"
server_key_iv: "3222665a4c93c18e4e14cfde3b9aeed8"
server_key_version: "encryptionkey@0"
I20260620 18:47:23.958818 28166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "49ccd525618543ad97e0e5a581631fae"
format_stamp: "Formatted at 2026-06-20 18:47:23 on dist-test-slave-k6lg"
server_key: "180c1f90710aa8bdecec29d3cceaec4e"
server_key_iv: "3222665a4c93c18e4e14cfde3b9aeed8"
server_key_version: "encryptionkey@0"
I20260620 18:47:23.964056 28166 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:47:23.967469 28181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:23.968866 28166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:47:23.969077 28166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "49ccd525618543ad97e0e5a581631fae"
format_stamp: "Formatted at 2026-06-20 18:47:23 on dist-test-slave-k6lg"
server_key: "180c1f90710aa8bdecec29d3cceaec4e"
server_key_iv: "3222665a4c93c18e4e14cfde3b9aeed8"
server_key_version: "encryptionkey@0"
I20260620 18:47:23.969303 28166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:23.989475 28166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:23.993805 28166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:23.994213 28166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:24.013351 28166 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:37665
I20260620 18:47:24.013360 28232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:37665 every 8 connection(s)
I20260620 18:47:24.014802 28166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:47:24.018639 28233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:47:24.019964 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 28166
I20260620 18:47:24.020133 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:47:24.020694 6022 external_mini_cluster.cc:1428] Setting key 322635ba5b208297c6c603f9e6c0c664
I20260620 18:47:24.033497 28233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49ccd525618543ad97e0e5a581631fae: Bootstrap starting.
I20260620 18:47:24.035176 28236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:24.022585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41307 (local address 127.5.225.190:37665)
0620 18:47:24.024325 (+ 1740us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:24.024337 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:24.024385 (+ 48us) server_negotiation.cc:407] Connection header received
0620 18:47:24.025987 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:24.026011 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:24.026528 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:24.026879 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:24.027470 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:24.028743 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:24.030687 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:24.031393 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:24.031529 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:24.032798 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:24.032805 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:24.032819 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:47:24.033355 (+ 536us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:24.033430 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:47:24.033456 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:24.033554 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:24.033837 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 18:47:24.034119 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1490,"thread_start_us":1251,"threads_started":1}
I20260620 18:47:24.036841 28233 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49ccd525618543ad97e0e5a581631fae: Neither blocks nor log segments found. Creating new log.
I20260620 18:47:24.038021 28233 log.cc:824] T 00000000000000000000000000000000 P 49ccd525618543ad97e0e5a581631fae: Log is configured to *not* fsync() on all Append() calls
I20260620 18:47:24.039005 6022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:47:24.041303 28233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49ccd525618543ad97e0e5a581631fae: No bootstrap required, opened a new log
I20260620 18:47:24.050513 28236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:24.044023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49201 (local address 127.5.225.190:37665)
0620 18:47:24.044212 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:24.044218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:24.044716 (+ 498us) server_negotiation.cc:407] Connection header received
0620 18:47:24.044838 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:24.044843 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:24.044920 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:24.045028 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:24.045485 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:24.046621 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:24.047876 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:24.048608 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:24.048743 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:24.049754 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:24.049760 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:24.049765 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:47:24.049786 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:24.049829 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:47:24.049847 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:24.049941 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:24.050066 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 18:47:24.050158 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:47:24.051484 28233 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49ccd525618543ad97e0e5a581631fae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ccd525618543ad97e0e5a581631fae" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 37665 } }
I20260620 18:47:24.051939 28233 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49ccd525618543ad97e0e5a581631fae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:47:24.052052 28233 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49ccd525618543ad97e0e5a581631fae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49ccd525618543ad97e0e5a581631fae, State: Initialized, Role: FOLLOWER
I20260620 18:47:24.052597 28233 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49ccd525618543ad97e0e5a581631fae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ccd525618543ad97e0e5a581631fae" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 37665 } }
I20260620 18:47:24.052834 28233 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49ccd525618543ad97e0e5a581631fae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:47:24.052981 28233 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49ccd525618543ad97e0e5a581631fae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:47:24.053210 28233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49ccd525618543ad97e0e5a581631fae [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:47:24.055450 28233 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49ccd525618543ad97e0e5a581631fae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ccd525618543ad97e0e5a581631fae" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 37665 } }
I20260620 18:47:24.056077 28233 leader_election.cc:302] T 00000000000000000000000000000000 P 49ccd525618543ad97e0e5a581631fae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49ccd525618543ad97e0e5a581631fae; no voters:
I20260620 18:47:24.056627 28233 leader_election.cc:288] T 00000000000000000000000000000000 P 49ccd525618543ad97e0e5a581631fae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:47:24.056766 28243 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49ccd525618543ad97e0e5a581631fae [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:47:24.057446 28243 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49ccd525618543ad97e0e5a581631fae [term 1 LEADER]: Becoming Leader. State: Replica: 49ccd525618543ad97e0e5a581631fae, State: Running, Role: LEADER
I20260620 18:47:24.058151 28243 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49ccd525618543ad97e0e5a581631fae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ccd525618543ad97e0e5a581631fae" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 37665 } }
I20260620 18:47:24.058764 28233 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49ccd525618543ad97e0e5a581631fae [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:47:24.060974 28244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49ccd525618543ad97e0e5a581631fae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49ccd525618543ad97e0e5a581631fae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ccd525618543ad97e0e5a581631fae" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 37665 } } }
I20260620 18:47:24.060942 28245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49ccd525618543ad97e0e5a581631fae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49ccd525618543ad97e0e5a581631fae. Latest consensus state: current_term: 1 leader_uuid: "49ccd525618543ad97e0e5a581631fae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ccd525618543ad97e0e5a581631fae" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 37665 } } }
I20260620 18:47:24.061399 28244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49ccd525618543ad97e0e5a581631fae [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:24.061458 28245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49ccd525618543ad97e0e5a581631fae [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:24.065230 28248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:47:24.070353 28248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:47:24.079661 28248 catalog_manager.cc:1347] Generated new cluster ID: 23cde788ecae49c0b5d7dd6a83f6b649
I20260620 18:47:24.079828 28248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:47:24.099220 28248 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:47:24.100579 28248 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:47:24.106266 28248 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49ccd525618543ad97e0e5a581631fae: Generated new TSK 0
I20260620 18:47:24.107290 28248 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:47:24.113884 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 28166
W20260620 18:47:24.115044 28228 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41307
W20260620 18:47:24.115769 6022 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 28166: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:47:24.115926 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 28166
2026-06-20T18:47:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1907 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:47:24 dist-test-slave-k6lg krb5kdc[28264](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:47:24 dist-test-slave-k6lg krb5kdc[28264](info): set up 2 sockets
Jun 20 18:47:24 dist-test-slave-k6lg krb5kdc[28264](info): commencing operation
krb5kdc: starting...
W20260620 18:47:25.218617 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:47:25 dist-test-slave-k6lg krb5kdc[28264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:47:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:47:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
I20260620 18:47:25.473485 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:41165
--webserver_interface=127.5.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:38833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:41165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:47:25.812356 28277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:25.812695 28277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:25.812860 28277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:25.820122 28277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:47:25.820264 28277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:25.820367 28277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:25.820488 28277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:47:25.820565 28277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:47:25.830979 28277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:38833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:41165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:41165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:47:25.832401 28277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:25.833735 28277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:25.845758 28285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:25.845824 28283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:25.845788 28282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:25.846590 28277 server_base.cc:1034] running on GCE node
I20260620 18:47:25.847505 28277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:25.848984 28277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:25.850288 28277 hybrid_clock.cc:648] HybridClock initialized: now 1781981245850248 us; error 64 us; skew 500 ppm
I20260620 18:47:25.850673 28277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:25 dist-test-slave-k6lg krb5kdc[28264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981245, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:25.853859 28277 init.cc:377] Logged in from keytab as kudu/127.5.225.190@KRBTEST.COM (short username kudu)
I20260620 18:47:25.855558 28277 webserver.cc:460] Webserver started at http://127.5.225.190:40627/ using document root <none> and password file <none>
I20260620 18:47:25.856333 28277 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:25.856477 28277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:25.856758 28277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:47:25.859800 28277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "9234d8dc0f0a43debdcb0c853b3bca6c"
format_stamp: "Formatted at 2026-06-20 18:47:25 on dist-test-slave-k6lg"
server_key: "bda1c454a810c7e59f3d232e44da1c55"
server_key_iv: "cd92cef1e3579f0ddbe09486e7881b49"
server_key_version: "encryptionkey@0"
I20260620 18:47:25.860530 28277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "9234d8dc0f0a43debdcb0c853b3bca6c"
format_stamp: "Formatted at 2026-06-20 18:47:25 on dist-test-slave-k6lg"
server_key: "bda1c454a810c7e59f3d232e44da1c55"
server_key_iv: "cd92cef1e3579f0ddbe09486e7881b49"
server_key_version: "encryptionkey@0"
I20260620 18:47:25.865620 28277 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260620 18:47:25.869270 28292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:25.870898 28277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:47:25.871104 28277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "9234d8dc0f0a43debdcb0c853b3bca6c"
format_stamp: "Formatted at 2026-06-20 18:47:25 on dist-test-slave-k6lg"
server_key: "bda1c454a810c7e59f3d232e44da1c55"
server_key_iv: "cd92cef1e3579f0ddbe09486e7881b49"
server_key_version: "encryptionkey@0"
I20260620 18:47:25.871318 28277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:25.876963 28277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:25.881254 28277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:25.881553 28277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:25.900306 28277 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:41165
I20260620 18:47:25.900338 28343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:41165 every 8 connection(s)
I20260620 18:47:25.901708 28277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:47:25.905687 28344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:47:25.907542 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 28277
I20260620 18:47:25.907732 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:47:25.908152 6022 external_mini_cluster.cc:1428] Setting key 978bee7e823aedcfb51709046ef0367f
I20260620 18:47:25.916540 28347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:25.909927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55553 (local address 127.5.225.190:41165)
0620 18:47:25.910933 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:25.910948 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:25.911001 (+ 53us) server_negotiation.cc:407] Connection header received
0620 18:47:25.912740 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:25.912772 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:25.913225 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:25.913688 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:25.914639 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:25.914647 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:25.914663 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:47:25.914680 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:25.914746 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:47:25.914774 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:25.914990 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:25.915126 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 18:47:25.915316 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":740,"thread_start_us":499,"threads_started":1}
I20260620 18:47:25.920588 6022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:47:25.920653 28344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9234d8dc0f0a43debdcb0c853b3bca6c: Bootstrap starting.
I20260620 18:47:25.924446 28344 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9234d8dc0f0a43debdcb0c853b3bca6c: Neither blocks nor log segments found. Creating new log.
I20260620 18:47:25.925757 28344 log.cc:824] T 00000000000000000000000000000000 P 9234d8dc0f0a43debdcb0c853b3bca6c: Log is configured to *not* fsync() on all Append() calls
I20260620 18:47:25.928333 28347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:25.925637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36797 (local address 127.5.225.190:41165)
0620 18:47:25.925853 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:25.925862 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:25.926361 (+ 499us) server_negotiation.cc:407] Connection header received
0620 18:47:25.926582 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:25.926591 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:25.926652 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:25.926774 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:25.927686 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:25.927690 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:25.927693 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:47:25.927707 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:25.927742 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:47:25.927755 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:25.927840 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:25.927979 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 18:47:25.928023 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:47:25.929246 28344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9234d8dc0f0a43debdcb0c853b3bca6c: No bootstrap required, opened a new log
I20260620 18:47:25.936556 28344 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9234d8dc0f0a43debdcb0c853b3bca6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9234d8dc0f0a43debdcb0c853b3bca6c" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 41165 } }
I20260620 18:47:25.936888 28344 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9234d8dc0f0a43debdcb0c853b3bca6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:47:25.936971 28344 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9234d8dc0f0a43debdcb0c853b3bca6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9234d8dc0f0a43debdcb0c853b3bca6c, State: Initialized, Role: FOLLOWER
I20260620 18:47:25.937451 28344 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9234d8dc0f0a43debdcb0c853b3bca6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9234d8dc0f0a43debdcb0c853b3bca6c" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 41165 } }
I20260620 18:47:25.937647 28344 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9234d8dc0f0a43debdcb0c853b3bca6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:47:25.937745 28344 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9234d8dc0f0a43debdcb0c853b3bca6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:47:25.937865 28344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9234d8dc0f0a43debdcb0c853b3bca6c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:47:25.940030 28344 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9234d8dc0f0a43debdcb0c853b3bca6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9234d8dc0f0a43debdcb0c853b3bca6c" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 41165 } }
I20260620 18:47:25.940454 28344 leader_election.cc:302] T 00000000000000000000000000000000 P 9234d8dc0f0a43debdcb0c853b3bca6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9234d8dc0f0a43debdcb0c853b3bca6c; no voters:
I20260620 18:47:25.940970 28344 leader_election.cc:288] T 00000000000000000000000000000000 P 9234d8dc0f0a43debdcb0c853b3bca6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:47:25.941128 28354 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9234d8dc0f0a43debdcb0c853b3bca6c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:47:25.942346 28354 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9234d8dc0f0a43debdcb0c853b3bca6c [term 1 LEADER]: Becoming Leader. State: Replica: 9234d8dc0f0a43debdcb0c853b3bca6c, State: Running, Role: LEADER
I20260620 18:47:25.942837 28354 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9234d8dc0f0a43debdcb0c853b3bca6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9234d8dc0f0a43debdcb0c853b3bca6c" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 41165 } }
I20260620 18:47:25.943156 28344 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9234d8dc0f0a43debdcb0c853b3bca6c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:47:25.945489 28355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9234d8dc0f0a43debdcb0c853b3bca6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9234d8dc0f0a43debdcb0c853b3bca6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9234d8dc0f0a43debdcb0c853b3bca6c" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 41165 } } }
I20260620 18:47:25.945587 28356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9234d8dc0f0a43debdcb0c853b3bca6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9234d8dc0f0a43debdcb0c853b3bca6c. Latest consensus state: current_term: 1 leader_uuid: "9234d8dc0f0a43debdcb0c853b3bca6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9234d8dc0f0a43debdcb0c853b3bca6c" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 41165 } } }
I20260620 18:47:25.945873 28355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9234d8dc0f0a43debdcb0c853b3bca6c [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:25.945919 28356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9234d8dc0f0a43debdcb0c853b3bca6c [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:25.947404 28359 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:47:25.952840 28359 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:47:25.961337 28359 catalog_manager.cc:1347] Generated new cluster ID: 4bf78645c85f44cda22427d7d0df3721
I20260620 18:47:25.961440 28359 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:47:25.988270 28359 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:47:25.988765 28359 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:47:25.998113 28359 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9234d8dc0f0a43debdcb0c853b3bca6c: Generated new TSK 0
I20260620 18:47:25.999091 28359 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:47:26.018283 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 28277
W20260620 18:47:26.019383 28339 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55553
W20260620 18:47:26.020159 6022 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 28277: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:47:26.020275 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 28277
2026-06-20T18:47:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1901 ms)
[----------] 18 tests from AuthTokenIssuingTest (34832 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:47:26 dist-test-slave-k6lg krb5kdc[28375](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:47:26 dist-test-slave-k6lg krb5kdc[28375](info): set up 2 sockets
Jun 20 18:47:26 dist-test-slave-k6lg krb5kdc[28375](info): commencing operation
krb5kdc: starting...
W20260620 18:47:27.134876 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:47:27 dist-test-slave-k6lg krb5kdc[28375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:47:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:47:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
I20260620 18:47:27.398307 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:39507
--webserver_interface=127.5.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:35673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:39507,127.5.225.189:40581,127.5.225.188:40777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:47:27.735421 28388 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:47:27.735814 28388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:27.735956 28388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:27.736094 28388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:27.743569 28388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:47:27.743690 28388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:27.743752 28388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:27.743866 28388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:47:27.743956 28388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:47:27.754467 28388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:35673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:39507,127.5.225.189:40581,127.5.225.188:40777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:39507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:47:27.755913 28388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:27.757340 28388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:27.769785 28393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:27.769826 28394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:27.769861 28396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:27.771330 28388 server_base.cc:1034] running on GCE node
I20260620 18:47:27.772140 28388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:27.773627 28388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:27.774858 28388 hybrid_clock.cc:648] HybridClock initialized: now 1781981247774764 us; error 57 us; skew 500 ppm
I20260620 18:47:27.775219 28388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:27 dist-test-slave-k6lg krb5kdc[28375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981247, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:27.782382 28388 init.cc:377] Logged in from keytab as kudu/127.5.225.190@KRBTEST.COM (short username kudu)
I20260620 18:47:27.784145 28388 webserver.cc:460] Webserver started at http://127.5.225.190:37451/ using document root <none> and password file <none>
I20260620 18:47:27.784984 28388 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:27.785121 28388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:27.785478 28388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:47:27.788811 28388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "8d2a742026ac4e35b5221e632a8e08ba"
format_stamp: "Formatted at 2026-06-20 18:47:27 on dist-test-slave-k6lg"
server_key: "b34414bdb824fb1e4529acd4fccc239d"
server_key_iv: "f89d491e3e4aeda24f25359a55576ffb"
server_key_version: "encryptionkey@0"
I20260620 18:47:27.789639 28388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "8d2a742026ac4e35b5221e632a8e08ba"
format_stamp: "Formatted at 2026-06-20 18:47:27 on dist-test-slave-k6lg"
server_key: "b34414bdb824fb1e4529acd4fccc239d"
server_key_iv: "f89d491e3e4aeda24f25359a55576ffb"
server_key_version: "encryptionkey@0"
I20260620 18:47:27.795194 28388 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 18:47:27.798755 28403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:27.800194 28388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:47:27.800398 28388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "8d2a742026ac4e35b5221e632a8e08ba"
format_stamp: "Formatted at 2026-06-20 18:47:27 on dist-test-slave-k6lg"
server_key: "b34414bdb824fb1e4529acd4fccc239d"
server_key_iv: "f89d491e3e4aeda24f25359a55576ffb"
server_key_version: "encryptionkey@0"
I20260620 18:47:27.800653 28388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:27.823377 28388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:27.828269 28388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:27.828573 28388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:27.848114 28388 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:39507
I20260620 18:47:27.848165 28454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:39507 every 8 connection(s)
I20260620 18:47:27.849538 28388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:47:27.853871 28455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:47:27.856622 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 28388
I20260620 18:47:27.856854 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:47:27.857465 6022 external_mini_cluster.cc:1428] Setting key 996e3e97920ed1346f0386fed6e609b7
I20260620 18:47:27.861565 28455 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39507 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:47:27 dist-test-slave-k6lg krb5kdc[28375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981247, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.225.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.189@KRBTEST.COM" created.
I20260620 18:47:27.888475 28459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:27.865255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49954 (local address 127.5.225.190:39507)
0620 18:47:27.866653 (+ 1398us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:27.866659 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:27.866699 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:47:27.869488 (+ 2789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:27.869494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:27.869739 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:27.869924 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:27.871621 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:27.872836 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:27.875130 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:27.875704 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:27.875833 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:27.880058 (+ 4225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:27.880072 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:27.880092 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:27.880155 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:27.884170 (+ 4015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:27.884944 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:27.884956 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:27.884961 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:27.885121 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:27.885553 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:27.885556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:27.885558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:27.886228 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:27.886448 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:27.887123 (+ 675us) server_negotiation.cc:299] Negotiation successful
0620 18:47:27.887429 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":1211,"thread_start_us":1006,"threads_started":1}
I20260620 18:47:27.888518 28458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:27.865253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39507 (local address 127.0.0.1:49954)
0620 18:47:27.866197 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:27.866255 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:27.869056 (+ 2801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:27.870026 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:27.870036 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:27.870904 (+ 868us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:47:27.871398 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:27.871424 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:27.873002 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:27.873010 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:27.874968 (+ 1958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:27.874987 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:27.875953 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:27.875964 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:27.876191 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:27.876934 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:47:27.876973 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:47:27.879792 (+ 2819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:47:27.884340 (+ 4548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:27.884351 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:27.884361 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:27.884784 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:27.885278 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:27.885282 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:27.885285 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:27.885445 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:27.886362 (+ 917us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:47:27.886373 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:47:27.886671 (+ 298us) client_negotiation.cc:769] Sending connection context
0620 18:47:27.887003 (+ 332us) client_negotiation.cc:240] Negotiation successful
0620 18:47:27.887646 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":80,"thread_start_us":408,"threads_started":1}
I20260620 18:47:27.892637 28455 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.225.189" port: 40581 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:47:27.893990 28458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:27.893426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:40581 (local address 127.0.0.1:60252)
0620 18:47:27.893654 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:27.893805 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.189:40581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260620 18:47:27.894307 28406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.225.189:40581: connect: Connection refused (error 111)
I20260620 18:47:27.896072 28458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:27.895602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:40581 (local address 127.0.0.1:60260)
0620 18:47:27.895769 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:27.895871 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.189:40581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 18:47:27.896339 28455 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.225.189:40581: Network error: Client connection negotiation failed: client connection to 127.5.225.189:40581: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.189.keytab.
Entry for principal kudu/127.5.225.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.189.keytab.
I20260620 18:47:27.921830 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.189:40581
--webserver_interface=127.5.225.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.189.keytab
--principal=kudu/127.5.225.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:35673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:39507,127.5.225.189:40581,127.5.225.188:40777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:47:27.945710 28455 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.225.189" port: 40581 } attempt: 1
I20260620 18:47:27.947136 28458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:27.946446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:40581 (local address 127.0.0.1:60272)
0620 18:47:27.946738 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:27.946944 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.189:40581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260620 18:47:27.948503 28458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:27.948007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:40581 (local address 127.0.0.1:60276)
0620 18:47:27.948200 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:27.948313 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.189:40581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 18:47:27.948722 28455 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.225.189:40581: Network error: Client connection negotiation failed: client connection to 127.5.225.189:40581: connect: Connection refused (error 111)
I20260620 18:47:28.016993 28455 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.225.189" port: 40581 } attempt: 2
I20260620 18:47:28.018365 28458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:28.017770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:40581 (local address 127.0.0.1:60280)
0620 18:47:28.018015 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:28.018132 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.189:40581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 18:47:28.019487 28458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:28.019060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:40581 (local address 127.0.0.1:60288)
0620 18:47:28.019207 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:28.019303 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.189:40581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 18:47:28.019718 28455 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.225.189:40581: Network error: Client connection negotiation failed: client connection to 127.5.225.189:40581: connect: Connection refused (error 111)
I20260620 18:47:28.111009 28455 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.225.189" port: 40581 } attempt: 3
I20260620 18:47:28.112421 28458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:28.111751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:40581 (local address 127.0.0.1:60298)
0620 18:47:28.112043 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:28.112197 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.189:40581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260620 18:47:28.113843 28458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:28.113264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:40581 (local address 127.0.0.1:60314)
0620 18:47:28.113474 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:28.113611 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.189:40581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 18:47:28.114146 28455 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.225.189:40581: Network error: Client connection negotiation failed: client connection to 127.5.225.189:40581: connect: Connection refused (error 111)
I20260620 18:47:28.257534 28455 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.225.189" port: 40581 } attempt: 4
I20260620 18:47:28.258877 28458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:28.258219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:40581 (local address 127.0.0.1:60320)
0620 18:47:28.258572 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:28.258688 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.189:40581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170}
I20260620 18:47:28.260118 28458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:28.259672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:40581 (local address 127.0.0.1:60326)
0620 18:47:28.259838 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:28.259949 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.189:40581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 18:47:28.260380 28455 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.225.189:40581: Network error: Client connection negotiation failed: client connection to 127.5.225.189:40581: connect: Connection refused (error 111)
W20260620 18:47:28.278745 28462 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:47:28.279141 28462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:28.279269 28462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:28.279412 28462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:28.287091 28462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:47:28.287189 28462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:28.287290 28462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:28.287350 28462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:47:28.287434 28462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:47:28.297703 28462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:35673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:39507,127.5.225.189:40581,127.5.225.188:40777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.189.keytab
--principal=kudu/127.5.225.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.189:40581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.225.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:47:28.299110 28462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:28.300362 28462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:28.312415 28467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:28.312443 28470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:28.312482 28468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:28.313868 28462 server_base.cc:1034] running on GCE node
I20260620 18:47:28.314702 28462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:28.316201 28462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:28.317446 28462 hybrid_clock.cc:648] HybridClock initialized: now 1781981248317361 us; error 79 us; skew 500 ppm
I20260620 18:47:28.317780 28462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:28 dist-test-slave-k6lg krb5kdc[28375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981248, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:28.321060 28462 init.cc:377] Logged in from keytab as kudu/127.5.225.189@KRBTEST.COM (short username kudu)
I20260620 18:47:28.322604 28462 webserver.cc:460] Webserver started at http://127.5.225.189:39281/ using document root <none> and password file <none>
I20260620 18:47:28.323262 28462 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:28.323350 28462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:28.323603 28462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:47:28.326625 28462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-1/data/instance:
uuid: "185079f219cb40858b2244200f74337e"
format_stamp: "Formatted at 2026-06-20 18:47:28 on dist-test-slave-k6lg"
server_key: "0649aebb269f6756f206d42deeea32d0"
server_key_iv: "bdc3dead004d4e98219430d070d02f4e"
server_key_version: "encryptionkey@0"
I20260620 18:47:28.327378 28462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-1/wal/instance:
uuid: "185079f219cb40858b2244200f74337e"
format_stamp: "Formatted at 2026-06-20 18:47:28 on dist-test-slave-k6lg"
server_key: "0649aebb269f6756f206d42deeea32d0"
server_key_iv: "bdc3dead004d4e98219430d070d02f4e"
server_key_version: "encryptionkey@0"
I20260620 18:47:28.332805 28462 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 18:47:28.336287 28477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:28.337754 28462 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:47:28.337961 28462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-1/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-1/wal
uuid: "185079f219cb40858b2244200f74337e"
format_stamp: "Formatted at 2026-06-20 18:47:28 on dist-test-slave-k6lg"
server_key: "0649aebb269f6756f206d42deeea32d0"
server_key_iv: "bdc3dead004d4e98219430d070d02f4e"
server_key_version: "encryptionkey@0"
I20260620 18:47:28.338240 28462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:28.361627 28462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:28.366413 28462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:28.366703 28462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:28.385563 28462 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.189:40581
I20260620 18:47:28.385582 28528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.189:40581 every 8 connection(s)
I20260620 18:47:28.387007 28462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-1/data/info.pb
I20260620 18:47:28.389936 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 28462
I20260620 18:47:28.390201 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-1/wal/instance
I20260620 18:47:28.390625 6022 external_mini_cluster.cc:1428] Setting key 2c6384910cb54d7cd82cfe07c4c018fa
I20260620 18:47:28.391058 28529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:47:28.397608 28529 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39507 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:47:28 dist-test-slave-k6lg krb5kdc[28375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981248, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.189@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.225.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.188@KRBTEST.COM" created.
I20260620 18:47:28.420135 28532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:28.400260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49962 (local address 127.5.225.190:39507)
0620 18:47:28.400748 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:28.400753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:28.401770 (+ 1017us) server_negotiation.cc:407] Connection header received
0620 18:47:28.403712 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:28.403717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:28.403818 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:28.403939 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:28.406015 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:28.407194 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:28.409056 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:28.409472 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:28.409611 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:28.413412 (+ 3801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:28.413417 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:28.413420 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:28.413465 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:28.416529 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:28.417470 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:28.417477 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:28.417483 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:28.417638 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:28.418029 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:28.418037 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:28.418043 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:28.418527 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:28.418690 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:28.419382 (+ 692us) server_negotiation.cc:299] Negotiation successful
0620 18:47:28.419644 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":261,"threads_started":1}
I20260620 18:47:28.420770 28533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:28.400867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39507 (local address 127.0.0.1:49962)
0620 18:47:28.401585 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:28.401622 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:28.403413 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:28.404207 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:28.404216 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:28.404789 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:47:28.405740 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:28.405769 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:28.407397 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:28.407402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:28.408900 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:28.408912 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:28.409715 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:28.409722 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:28.409981 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:28.410748 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:47:28.410776 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:47:28.413175 (+ 2399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:47:28.416677 (+ 3502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:28.416684 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:28.416698 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:28.417339 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:28.417751 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:28.417755 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:28.417757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:28.417919 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:28.418649 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:47:28.418655 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:47:28.418984 (+ 329us) client_negotiation.cc:769] Sending connection context
0620 18:47:28.419249 (+ 265us) client_negotiation.cc:240] Negotiation successful
0620 18:47:28.419635 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":292,"threads_started":1}
I20260620 18:47:28.422680 28529 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.225.189" port: 40581 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:47:28 dist-test-slave-k6lg krb5kdc[28375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981248, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.189@KRBTEST.COM for kudu/127.5.225.189@KRBTEST.COM
I20260620 18:47:28.439035 28533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:28.423510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:40581 (local address 127.0.0.1:60330)
0620 18:47:28.423705 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:28.423728 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:28.423873 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:28.424751 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:28.424754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:28.425106 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:47:28.425248 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:28.425256 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:28.426913 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:28.426919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:28.427922 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:28.427933 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:28.429171 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:28.429175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:28.429331 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:28.429933 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:47:28.429962 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:47:28.432014 (+ 2052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:47:28.436498 (+ 4484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:28.436502 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:28.436504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:28.436850 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:28.437274 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:28.437278 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:28.437280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:28.437416 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:28.438269 (+ 853us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:47:28.438273 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:47:28.438394 (+ 121us) client_negotiation.cc:769] Sending connection context
0620 18:47:28.438522 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 18:47:28.438749 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":2}
I20260620 18:47:28.439529 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:28.423537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60330 (local address 127.5.225.189:40581)
0620 18:47:28.424218 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:28.424228 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:28.424262 (+ 34us) server_negotiation.cc:407] Connection header received
0620 18:47:28.424320 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:28.424330 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:28.424582 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:28.424736 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:28.425416 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:28.426769 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:28.428060 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:28.429034 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:28.429182 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:28.432232 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:28.432248 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:28.432267 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:28.432315 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:28.436344 (+ 4029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:28.436984 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:28.436997 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:28.437003 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:28.437165 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:28.437526 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:28.437533 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:28.437539 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:28.438054 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:28.438318 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:28.438720 (+ 402us) server_negotiation.cc:299] Negotiation successful
0620 18:47:28.439017 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":479,"thread_start_us":299,"threads_started":1}
I20260620 18:47:28.442608 28529 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40777 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:47:28.443907 28533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:28.443334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:40777 (local address 127.0.0.1:59074)
0620 18:47:28.443563 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:28.443693 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260620 18:47:28.444223 28481 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40777: connect: Connection refused (error 111)
I20260620 18:47:28.445794 28533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:28.445434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:40777 (local address 127.0.0.1:59078)
0620 18:47:28.445615 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:28.445697 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260620 18:47:28.446020 28529 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.225.188:40777: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40777: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.188.keytab.
Entry for principal kudu/127.5.225.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.188.keytab.
I20260620 18:47:28.453312 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.188:40777
--webserver_interface=127.5.225.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.188.keytab
--principal=kudu/127.5.225.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:35673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:39507,127.5.225.189:40581,127.5.225.188:40777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:47:28.495474 28529 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40777 } attempt: 1
I20260620 18:47:28.496697 28533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:28.496221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:40777 (local address 127.0.0.1:59084)
0620 18:47:28.496418 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:28.496543 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 18:47:28.497833 28533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:28.497492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:40777 (local address 127.0.0.1:59098)
0620 18:47:28.497648 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:28.497724 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 18:47:28.498070 28529 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.225.188:40777: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40777: connect: Connection refused (error 111)
I20260620 18:47:28.559711 28455 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.225.189" port: 40581 } attempt: 5
I20260620 18:47:28.566295 28529 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40777 } attempt: 2
I20260620 18:47:28.567303 28533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:28.566849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:40777 (local address 127.0.0.1:59106)
0620 18:47:28.567069 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:28.567150 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 18:47:28.568429 28533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:28.568010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:40777 (local address 127.0.0.1:59114)
0620 18:47:28.568186 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:28.568271 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
Jun 20 18:47:28 dist-test-slave-k6lg krb5kdc[28375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981247, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for kudu/127.5.225.189@KRBTEST.COM
W20260620 18:47:28.568717 28529 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.225.188:40777: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40777: connect: Connection refused (error 111)
I20260620 18:47:28.574819 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:28.560612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60340 (local address 127.5.225.189:40581)
0620 18:47:28.560806 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:28.560811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:28.560829 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:47:28.561099 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:28.561104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:28.561174 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:28.561270 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:28.562318 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:28.563037 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:28.565039 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:28.565495 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:28.565592 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:28.569469 (+ 3877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:28.569482 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:28.569485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:28.569517 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:28.572195 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:28.573031 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:28.573035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:28.573037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:28.573156 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:28.573569 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:28.573573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:28.573575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:28.573853 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:28.573997 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:28.574461 (+ 464us) server_negotiation.cc:299] Negotiation successful
0620 18:47:28.574624 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:47:28.574945 28458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:28.560463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:40581 (local address 127.0.0.1:60340)
0620 18:47:28.560724 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:28.560748 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:28.560888 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:28.561341 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:28.561346 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:28.561882 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:47:28.562127 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:28.562139 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:28.563192 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:28.563200 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:28.564861 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:28.564878 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:28.565660 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:28.565675 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:28.565899 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:28.566730 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:47:28.566753 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:47:28.569292 (+ 2539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:47:28.572329 (+ 3037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:28.572335 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:28.572339 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:28.572865 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:28.573266 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:28.573271 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:28.573275 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:28.573432 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:28.573985 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:47:28.573990 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:47:28.574154 (+ 164us) client_negotiation.cc:769] Sending connection context
0620 18:47:28.574356 (+ 202us) client_negotiation.cc:240] Negotiation successful
0620 18:47:28.574620 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 18:47:28.575970 28455 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40777 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:47:28.577030 28458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:28.576610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:40777 (local address 127.0.0.1:59116)
0620 18:47:28.576787 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:28.576875 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 18:47:28.577962 28458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:28.577629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:40777 (local address 127.0.0.1:59118)
0620 18:47:28.577766 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:28.577855 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 18:47:28.578207 28455 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.225.188:40777: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40777: connect: Connection refused (error 111)
I20260620 18:47:28.629472 28455 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40777 } attempt: 1
I20260620 18:47:28.631067 28458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:28.630394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:40777 (local address 127.0.0.1:59122)
0620 18:47:28.630681 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:28.630818 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260620 18:47:28.632251 28458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:28.631871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:40777 (local address 127.0.0.1:59132)
0620 18:47:28.632045 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:28.632126 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 18:47:28.632508 28455 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.225.188:40777: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40777: connect: Connection refused (error 111)
I20260620 18:47:28.659948 28529 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40777 } attempt: 3
I20260620 18:47:28.661283 28533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:28.660647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:40777 (local address 127.0.0.1:59146)
0620 18:47:28.660909 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:28.661044 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 18:47:28.662572 28533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:28.661943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:40777 (local address 127.0.0.1:59148)
0620 18:47:28.662280 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:28.662395 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185}
W20260620 18:47:28.662843 28529 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.225.188:40777: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40777: connect: Connection refused (error 111)
I20260620 18:47:28.700774 28455 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40777 } attempt: 2
I20260620 18:47:28.702080 28458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:28.701472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:40777 (local address 127.0.0.1:59158)
0620 18:47:28.701758 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:28.701872 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260620 18:47:28.703233 28458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:28.702858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:40777 (local address 127.0.0.1:59170)
0620 18:47:28.703023 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:28.703109 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 18:47:28.703518 28455 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.225.188:40777: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40777: connect: Connection refused (error 111)
W20260620 18:47:28.786866 28537 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:47:28.787246 28537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:28.787374 28537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:28.787514 28537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:28.794643 28537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:47:28.794772 28537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:28.794883 28537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:28.794972 28537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:47:28.795061 28537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:47:28.804976 28537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:35673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:39507,127.5.225.189:40581,127.5.225.188:40777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.188.keytab
--principal=kudu/127.5.225.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.188:40777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.225.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:47:28.806104 28529 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40777 } attempt: 4
I20260620 18:47:28.806298 28537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:28.807327 28533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:28.806837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:40777 (local address 127.0.0.1:59184)
0620 18:47:28.807083 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:28.807181 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 18:47:28.808017 28537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 18:47:28.808310 28533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:28.807989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:40777 (local address 127.0.0.1:59198)
0620 18:47:28.808140 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:28.808213 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 18:47:28.808522 28529 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.225.188:40777: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40777: connect: Connection refused (error 111)
I20260620 18:47:28.809721 28455 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40777 } attempt: 3
I20260620 18:47:28.810787 28458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:28.810292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:40777 (local address 127.0.0.1:59208)
0620 18:47:28.810481 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:28.810579 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 18:47:28.811928 28458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:28.811506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:40777 (local address 127.0.0.1:59214)
0620 18:47:28.811709 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:28.811799 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260620 18:47:28.812161 28455 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.225.188:40777: Network error: Client connection negotiation failed: client connection to 127.5.225.188:40777: connect: Connection refused (error 111)
W20260620 18:47:28.820403 28542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:28.820446 28543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:28.820626 28545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:28.823858 28537 server_base.cc:1034] running on GCE node
I20260620 18:47:28.824806 28537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:28.826417 28537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:28.827687 28537 hybrid_clock.cc:648] HybridClock initialized: now 1781981248827619 us; error 54 us; skew 500 ppm
I20260620 18:47:28.828089 28537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:28 dist-test-slave-k6lg krb5kdc[28375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981248, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:28.831638 28537 init.cc:377] Logged in from keytab as kudu/127.5.225.188@KRBTEST.COM (short username kudu)
I20260620 18:47:28.833221 28537 webserver.cc:460] Webserver started at http://127.5.225.188:35137/ using document root <none> and password file <none>
I20260620 18:47:28.833937 28537 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:28.834030 28537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:28.834476 28537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:47:28.837870 28537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-2/data/instance:
uuid: "2cd8338d53444eed8afe04c94aec0203"
format_stamp: "Formatted at 2026-06-20 18:47:28 on dist-test-slave-k6lg"
server_key: "11616e971cb21b6578e18175e6d007c8"
server_key_iv: "31afcd92ec7a843368205b82fb747f2c"
server_key_version: "encryptionkey@0"
I20260620 18:47:28.838742 28537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-2/wal/instance:
uuid: "2cd8338d53444eed8afe04c94aec0203"
format_stamp: "Formatted at 2026-06-20 18:47:28 on dist-test-slave-k6lg"
server_key: "11616e971cb21b6578e18175e6d007c8"
server_key_iv: "31afcd92ec7a843368205b82fb747f2c"
server_key_version: "encryptionkey@0"
I20260620 18:47:28.844435 28537 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:47:28.848531 28552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:28.850112 28537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:47:28.850337 28537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-2/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-2/wal
uuid: "2cd8338d53444eed8afe04c94aec0203"
format_stamp: "Formatted at 2026-06-20 18:47:28 on dist-test-slave-k6lg"
server_key: "11616e971cb21b6578e18175e6d007c8"
server_key_iv: "31afcd92ec7a843368205b82fb747f2c"
server_key_version: "encryptionkey@0"
I20260620 18:47:28.850576 28537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:28.884352 28537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:28.888617 28537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:28.888908 28537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:28.906853 28537 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.188:40777
I20260620 18:47:28.906873 28603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.188:40777 every 8 connection(s)
I20260620 18:47:28.908536 28537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-2/data/info.pb
I20260620 18:47:28.912220 28604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:47:28.918833 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 28537
I20260620 18:47:28.919016 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980894474392-6022-0/minicluster-data/master-2/wal/instance
I20260620 18:47:28.918462 28604 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39507 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:47:28.919499 6022 external_mini_cluster.cc:1428] Setting key 3b4b44bd3698314f52cbab5fccfa2de2
Jun 20 18:47:28 dist-test-slave-k6lg krb5kdc[28375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981248, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.188@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
Jun 20 18:47:28 dist-test-slave-k6lg krb5kdc[28375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.188@KRBTEST.COM
I20260620 18:47:28.939762 28609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:28.922275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49970 (local address 127.5.225.190:39507)
0620 18:47:28.923549 (+ 1274us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:28.923554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:28.923570 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:47:28.924391 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:28.924396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:28.924495 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:28.924658 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:28.925908 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:28.926734 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:28.928347 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:28.928946 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:28.929037 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:28.932701 (+ 3664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:28.932711 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:28.932715 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:28.932755 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:28.936191 (+ 3436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:28.937038 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:28.937046 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:28.937050 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:28.937200 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:28.937818 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:28.937821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:28.937824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:28.938133 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:28.938330 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:28.939123 (+ 793us) server_negotiation.cc:299] Negotiation successful
0620 18:47:28.939352 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1176,"thread_start_us":1036,"threads_started":1}
I20260620 18:47:28.940825 28608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:28.922239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:39507 (local address 127.0.0.1:49970)
0620 18:47:28.922936 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:28.922983 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:28.924150 (+ 1167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:28.924746 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:28.924752 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:28.925401 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:47:28.925742 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:28.925758 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:28.926893 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:28.926898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:28.928202 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:28.928213 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:28.929150 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:28.929158 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:28.929418 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:28.930260 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:47:28.930280 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:47:28.932487 (+ 2207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:47:28.936356 (+ 3869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:28.936365 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:28.936375 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:28.936880 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:28.937413 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:28.937418 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:28.937421 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:28.937699 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:28.938293 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:47:28.938299 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:47:28.938753 (+ 454us) client_negotiation.cc:769] Sending connection context
0620 18:47:28.938994 (+ 241us) client_negotiation.cc:240] Negotiation successful
0620 18:47:28.939556 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":324,"threads_started":1}
I20260620 18:47:28.940843 28607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:28.921243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59226 (local address 127.5.225.188:40777)
0620 18:47:28.921891 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:28.921901 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:28.921938 (+ 37us) server_negotiation.cc:407] Connection header received
0620 18:47:28.924135 (+ 2197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:28.924156 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:28.924405 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:28.924558 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:28.925974 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:28.927393 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:28.928813 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:28.929481 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:28.929614 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:28.933718 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:28.933731 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:28.933749 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:28.933788 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:28.937250 (+ 3462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:28.938080 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:28.938090 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:28.938094 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:28.938285 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:28.938700 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:28.938704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:28.938706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:28.939064 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:28.939363 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:28.939673 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 18:47:28.939931 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":267,"threads_started":1}
I20260620 18:47:28.943204 28604 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.225.189" port: 40581 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:47:28 dist-test-slave-k6lg krb5kdc[28375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981248, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.188@KRBTEST.COM for kudu/127.5.225.189@KRBTEST.COM
Jun 20 18:47:28 dist-test-slave-k6lg krb5kdc[28375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:47:28.957367 28608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:28.943974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:40581 (local address 127.0.0.1:60354)
0620 18:47:28.944314 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:28.944331 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:28.944431 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:28.944765 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:28.944768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:28.945092 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:47:28.945249 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:28.945257 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:28.946431 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:28.946437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:28.947450 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:28.947463 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:28.948327 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:28.948334 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:28.948536 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:28.949164 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:47:28.949180 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:47:28.951311 (+ 2131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:47:28.954836 (+ 3525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:28.954843 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:28.954847 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:28.955261 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:28.955751 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:28.955755 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:28.955759 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:28.955907 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:28.956428 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:47:28.956433 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:47:28.956582 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 18:47:28.956720 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 18:47:28.956994 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219}
I20260620 18:47:28.957394 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:28.944132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60354 (local address 127.5.225.189:40581)
0620 18:47:28.944320 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:28.944326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:28.944381 (+ 55us) server_negotiation.cc:407] Connection header received
0620 18:47:28.944540 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:28.944545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:28.944620 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:28.944726 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:28.945393 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:28.946307 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:28.947586 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:28.948180 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:28.948270 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:28.951511 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:28.951518 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:28.951524 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:28.951570 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:28.954707 (+ 3137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:28.955426 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:28.955433 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:28.955437 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:28.955611 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:28.956021 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:28.956027 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:28.956030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:28.956311 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:28.956462 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:28.956808 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 18:47:28.957014 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 18:47:28.958398 28609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:28.943910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49982 (local address 127.5.225.190:39507)
0620 18:47:28.944146 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:28.944152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:28.944171 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:47:28.944315 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:28.944322 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:28.944415 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:28.944521 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:28.945842 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:28.946760 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:28.948087 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:28.948649 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:28.948796 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:28.952985 (+ 4189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:28.952990 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:28.952993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:28.953023 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:28.955854 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:28.956495 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:28.956501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:28.956504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:28.956629 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:28.956981 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:28.956987 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:28.956990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:28.957265 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:28.957639 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:28.957818 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 18:47:28.958037 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260620 18:47:28.959223 28604 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40777 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:47:28 dist-test-slave-k6lg krb5kdc[28375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981248, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.188@KRBTEST.COM for kudu/127.5.225.188@KRBTEST.COM
Jun 20 18:47:28 dist-test-slave-k6lg krb5kdc[28375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.189@KRBTEST.COM
I20260620 18:47:28.973544 28608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:28.960081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:40777 (local address 127.0.0.1:59234)
0620 18:47:28.960216 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:28.960236 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:28.960352 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:28.960680 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:28.960685 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:28.961124 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:47:28.961248 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:28.961256 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:28.962539 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:28.962544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:28.963699 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:28.963710 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:28.964592 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:28.964596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:28.964809 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:28.965403 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:47:28.965425 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:47:28.967771 (+ 2346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:47:28.971172 (+ 3401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:28.971177 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:28.971184 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:28.971687 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:28.972035 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:28.972043 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:28.972049 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:28.972208 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:28.972692 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:47:28.972699 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:47:28.972855 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 18:47:28.973021 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 18:47:28.973276 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 18:47:28.973892 28607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:28.960004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59234 (local address 127.5.225.188:40777)
0620 18:47:28.960218 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:28.960223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:28.960309 (+ 86us) server_negotiation.cc:407] Connection header received
0620 18:47:28.960457 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:28.960463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:28.960537 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:28.960629 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:28.961386 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:28.962430 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:28.963827 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:28.964453 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:28.964574 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:28.967964 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:28.967973 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:28.967979 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:28.968017 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:28.971048 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:28.971796 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:28.971800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:28.971802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:28.971927 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:28.972316 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:28.972319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:28.972321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:28.972568 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:28.972668 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:28.973278 (+ 610us) server_negotiation.cc:299] Negotiation successful
0620 18:47:28.973518 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260620 18:47:28.975859 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:28.960177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60366 (local address 127.5.225.189:40581)
0620 18:47:28.960344 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:28.960349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:28.960615 (+ 266us) server_negotiation.cc:407] Connection header received
0620 18:47:28.960723 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:28.960729 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:28.960799 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:28.960901 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:28.962272 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:28.963265 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:28.964781 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:28.965411 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:28.965543 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:28.970060 (+ 4517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:28.970064 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:28.970068 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:28.970096 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:28.973013 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:28.973830 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:28.973838 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:28.973844 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:28.974035 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:28.974472 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:28.974476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:28.974478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:28.974734 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:28.974991 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:28.975305 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 18:47:28.975527 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:47:28.979431 28604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cd8338d53444eed8afe04c94aec0203: Bootstrap starting.
I20260620 18:47:28.980561 6022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:47:28.983556 28604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cd8338d53444eed8afe04c94aec0203: Neither blocks nor log segments found. Creating new log.
I20260620 18:47:28.984622 28604 log.cc:824] T 00000000000000000000000000000000 P 2cd8338d53444eed8afe04c94aec0203: Log is configured to *not* fsync() on all Append() calls
I20260620 18:47:28.983944 28430 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "8d2a742026ac4e35b5221e632a8e08ba"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49982
I20260620 18:47:28.987457 28604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cd8338d53444eed8afe04c94aec0203: No bootstrap required, opened a new log
I20260620 18:47:28.989373 28455 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40777 } attempt: 4
I20260620 18:47:28.996261 28604 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cd8338d53444eed8afe04c94aec0203 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2a742026ac4e35b5221e632a8e08ba" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39507 } } peers { permanent_uuid: "185079f219cb40858b2244200f74337e" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 40581 } } peers { permanent_uuid: "2cd8338d53444eed8afe04c94aec0203" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40777 } }
I20260620 18:47:28.996598 28604 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cd8338d53444eed8afe04c94aec0203 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cd8338d53444eed8afe04c94aec0203, State: Initialized, Role: FOLLOWER
I20260620 18:47:28.997063 28604 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cd8338d53444eed8afe04c94aec0203 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2a742026ac4e35b5221e632a8e08ba" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39507 } } peers { permanent_uuid: "185079f219cb40858b2244200f74337e" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 40581 } } peers { permanent_uuid: "2cd8338d53444eed8afe04c94aec0203" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40777 } }
Jun 20 18:47:28 dist-test-slave-k6lg krb5kdc[28375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981247, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for kudu/127.5.225.188@KRBTEST.COM
I20260620 18:47:28.997987 28611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cd8338d53444eed8afe04c94aec0203 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2a742026ac4e35b5221e632a8e08ba" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39507 } } peers { permanent_uuid: "185079f219cb40858b2244200f74337e" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 40581 } } peers { permanent_uuid: "2cd8338d53444eed8afe04c94aec0203" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40777 } } }
I20260620 18:47:28.998373 28611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cd8338d53444eed8afe04c94aec0203 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:47:28.999233 28604 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cd8338d53444eed8afe04c94aec0203 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:47:29.007241 28458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:28.989938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:40777 (local address 127.0.0.1:59250)
0620 18:47:28.990152 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:28.990226 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:28.990339 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:28.990701 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:28.990706 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:28.991206 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:47:28.991409 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:28.991420 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:28.992879 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:28.992885 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:28.994383 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:28.994400 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:28.995444 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:28.995453 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:28.995664 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:28.996338 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:47:28.996359 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:47:28.998915 (+ 2556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:47:29.003883 (+ 4968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:29.003890 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:29.003895 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:29.004378 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:29.005049 (+ 671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:29.005055 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:29.005058 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:29.005260 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:29.006153 (+ 893us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:47:29.006177 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:47:29.006368 (+ 191us) client_negotiation.cc:769] Sending connection context
0620 18:47:29.006517 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 18:47:29.006819 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 18:47:29.007272 28607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:28.990132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59250 (local address 127.5.225.188:40777)
0620 18:47:28.990380 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:28.990386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:28.990407 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:47:28.990470 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:28.990477 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:28.990549 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:28.990653 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:28.991556 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:28.992725 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:28.994528 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:28.995298 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:28.995423 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:28.999088 (+ 3665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:28.999096 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:28.999109 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:28.999143 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:29.003708 (+ 4565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:29.004607 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:29.004615 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:29.004624 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:29.004852 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:29.005441 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:29.005447 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:29.005453 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:29.005992 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:29.006196 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:29.006665 (+ 469us) server_negotiation.cc:299] Negotiation successful
0620 18:47:29.006898 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260620 18:47:29.013690 28455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d2a742026ac4e35b5221e632a8e08ba: Bootstrap starting.
I20260620 18:47:29.017624 28455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d2a742026ac4e35b5221e632a8e08ba: Neither blocks nor log segments found. Creating new log.
I20260620 18:47:29.018926 28455 log.cc:824] T 00000000000000000000000000000000 P 8d2a742026ac4e35b5221e632a8e08ba: Log is configured to *not* fsync() on all Append() calls
W20260620 18:47:29.024843 28622 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2cd8338d53444eed8afe04c94aec0203: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:47:29.025017 28622 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:47:29.025113 28455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d2a742026ac4e35b5221e632a8e08ba: No bootstrap required, opened a new log
I20260620 18:47:29.009267 28537 master_runner.cc:186] Error getting master registration for 127.5.225.190:39507: OK, instance_id { permanent_uuid: "8d2a742026ac4e35b5221e632a8e08ba" instance_seqno: 1781981247842368 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:47:29.028041 28537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 18:47:29.033149 28455 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d2a742026ac4e35b5221e632a8e08ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2a742026ac4e35b5221e632a8e08ba" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39507 } } peers { permanent_uuid: "185079f219cb40858b2244200f74337e" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 40581 } } peers { permanent_uuid: "2cd8338d53444eed8afe04c94aec0203" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40777 } }
I20260620 18:47:29.033641 28455 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d2a742026ac4e35b5221e632a8e08ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d2a742026ac4e35b5221e632a8e08ba, State: Initialized, Role: FOLLOWER
I20260620 18:47:29.034111 28455 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d2a742026ac4e35b5221e632a8e08ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2a742026ac4e35b5221e632a8e08ba" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39507 } } peers { permanent_uuid: "185079f219cb40858b2244200f74337e" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 40581 } } peers { permanent_uuid: "2cd8338d53444eed8afe04c94aec0203" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40777 } }
I20260620 18:47:29.035195 28625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d2a742026ac4e35b5221e632a8e08ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2a742026ac4e35b5221e632a8e08ba" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39507 } } peers { permanent_uuid: "185079f219cb40858b2244200f74337e" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 40581 } } peers { permanent_uuid: "2cd8338d53444eed8afe04c94aec0203" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40777 } } }
I20260620 18:47:29.035579 28625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d2a742026ac4e35b5221e632a8e08ba [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:47:29.036351 28455 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d2a742026ac4e35b5221e632a8e08ba [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:47:29.036509 28430 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d2a742026ac4e35b5221e632a8e08ba [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 18:47:29.036746 28430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d2a742026ac4e35b5221e632a8e08ba [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:47:29.041010 28430 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d2a742026ac4e35b5221e632a8e08ba [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2a742026ac4e35b5221e632a8e08ba" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39507 } } peers { permanent_uuid: "185079f219cb40858b2244200f74337e" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 40581 } } peers { permanent_uuid: "2cd8338d53444eed8afe04c94aec0203" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40777 } }
I20260620 18:47:29.043109 28430 leader_election.cc:288] T 00000000000000000000000000000000 P 8d2a742026ac4e35b5221e632a8e08ba [CANDIDATE]: Term 1 election: Requested vote from peers 185079f219cb40858b2244200f74337e (127.5.225.189:40581), 2cd8338d53444eed8afe04c94aec0203 (127.5.225.188:40777)
I20260620 18:47:29.044023 28579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d2a742026ac4e35b5221e632a8e08ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2cd8338d53444eed8afe04c94aec0203"
I20260620 18:47:29.044524 28579 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cd8338d53444eed8afe04c94aec0203 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:47:29.047937 28579 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2cd8338d53444eed8afe04c94aec0203 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d2a742026ac4e35b5221e632a8e08ba in term 1.
I20260620 18:47:29.048724 28407 leader_election.cc:302] T 00000000000000000000000000000000 P 8d2a742026ac4e35b5221e632a8e08ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cd8338d53444eed8afe04c94aec0203, 8d2a742026ac4e35b5221e632a8e08ba; no voters:
I20260620 18:47:29.049330 28625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d2a742026ac4e35b5221e632a8e08ba [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:47:29.062317 28625 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d2a742026ac4e35b5221e632a8e08ba [term 1 LEADER]: Becoming Leader. State: Replica: 8d2a742026ac4e35b5221e632a8e08ba, State: Running, Role: LEADER
I20260620 18:47:29.063071 28625 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d2a742026ac4e35b5221e632a8e08ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2a742026ac4e35b5221e632a8e08ba" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39507 } } peers { permanent_uuid: "185079f219cb40858b2244200f74337e" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 40581 } } peers { permanent_uuid: "2cd8338d53444eed8afe04c94aec0203" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40777 } }
I20260620 18:47:29.071552 28631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d2a742026ac4e35b5221e632a8e08ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d2a742026ac4e35b5221e632a8e08ba. Latest consensus state: current_term: 1 leader_uuid: "8d2a742026ac4e35b5221e632a8e08ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2a742026ac4e35b5221e632a8e08ba" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39507 } } peers { permanent_uuid: "185079f219cb40858b2244200f74337e" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 40581 } } peers { permanent_uuid: "2cd8338d53444eed8afe04c94aec0203" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40777 } } }
I20260620 18:47:29.072072 28631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d2a742026ac4e35b5221e632a8e08ba [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:29.078696 28636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260620 18:47:29.081535 28409 tablet.cc:2369] T 00000000000000000000000000000000 P 8d2a742026ac4e35b5221e632a8e08ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:47:29.097995 28636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:47:29.108031 28529 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40777 } attempt: 5
Jun 20 18:47:29 dist-test-slave-k6lg krb5kdc[28375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981248, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.189@KRBTEST.COM for kudu/127.5.225.188@KRBTEST.COM
W20260620 18:47:29.141049 28558 tablet.cc:2369] T 00000000000000000000000000000000 P 2cd8338d53444eed8afe04c94aec0203: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:47:29.128358 28388 master_runner.cc:186] Error getting master registration for 127.5.225.189:40581: OK, instance_id { permanent_uuid: "185079f219cb40858b2244200f74337e" instance_seqno: 1781981248380218 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:47:29.147305 28388 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 18:47:29.149976 28533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:29.108745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:40777 (local address 127.0.0.1:59276)
0620 18:47:29.109109 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:29.109128 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:29.109235 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:29.123354 (+ 14119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:29.123360 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:29.123900 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:47:29.124087 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:29.124100 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:29.130300 (+ 6200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:29.130307 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:29.131892 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:29.131908 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:29.133849 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:29.133856 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:29.134136 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:29.134890 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:47:29.134920 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:47:29.138749 (+ 3829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:47:29.146391 (+ 7642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:29.146410 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:29.146415 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:29.146968 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:29.147720 (+ 752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:29.147727 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:29.147734 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:29.147916 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:29.148712 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:47:29.148717 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:47:29.148907 (+ 190us) client_negotiation.cc:769] Sending connection context
0620 18:47:29.149077 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 18:47:29.149463 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243}
I20260620 18:47:29.151479 28644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:29.109447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59276 (local address 127.5.225.188:40777)
0620 18:47:29.122416 (+ 12969us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:29.122422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:29.122544 (+ 122us) server_negotiation.cc:407] Connection header received
0620 18:47:29.122686 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:29.122784 (+ 98us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:29.123042 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:29.124343 (+ 1301us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:29.124773 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:29.126177 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:29.132201 (+ 6024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:29.133584 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:29.135276 (+ 1692us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:29.139121 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:29.139130 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:29.139136 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:29.139244 (+ 108us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:29.146140 (+ 6896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:29.147257 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:29.147265 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:29.147271 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:29.147538 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:29.148059 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:29.148065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:29.148069 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:29.148544 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:29.150441 (+ 1897us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:29.150619 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 18:47:29.150951 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12605,"thread_start_us":12224,"threads_started":1}
I20260620 18:47:29.152618 28579 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2cd8338d53444eed8afe04c94aec0203 [term 1 FOLLOWER]: Refusing update from remote peer 8d2a742026ac4e35b5221e632a8e08ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:47:29.156258 28607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:29.097982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59260 (local address 127.5.225.188:40777)
0620 18:47:29.098197 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:29.098203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:29.113074 (+ 14871us) server_negotiation.cc:407] Connection header received
0620 18:47:29.113502 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:29.113512 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:29.113620 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:29.113707 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:29.117149 (+ 3442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:29.118635 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:29.127002 (+ 8367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:29.127752 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:29.127851 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:29.148312 (+ 20461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:29.148323 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:29.148328 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:29.148367 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:29.152382 (+ 4015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:29.153418 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:29.153424 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:29.153428 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:29.153617 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:29.154260 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:29.154268 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:29.154272 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:29.154668 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:29.154954 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:29.155578 (+ 624us) server_negotiation.cc:299] Negotiation successful
0620 18:47:29.155860 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 18:47:29.159759 28631 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8d2a742026ac4e35b5221e632a8e08ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd8338d53444eed8afe04c94aec0203" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:47:29.164505 28609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:29.082938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49984 (local address 127.5.225.190:39507)
0620 18:47:29.083166 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:29.083175 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:29.104094 (+ 20919us) server_negotiation.cc:407] Connection header received
0620 18:47:29.104913 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:29.104922 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:29.105057 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:29.105144 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:29.128699 (+ 23555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:29.129876 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:29.146298 (+ 16422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:29.147052 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:29.147164 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:29.156501 (+ 9337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:29.156510 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:29.156515 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:29.156560 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:29.160767 (+ 4207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:29.161732 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:29.161742 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:29.161747 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:29.161905 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:29.162423 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:29.162429 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:29.162433 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:29.162825 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:29.163109 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:29.163769 (+ 660us) server_negotiation.cc:299] Negotiation successful
0620 18:47:29.164098 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 18:47:29.160858 28529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 185079f219cb40858b2244200f74337e: Bootstrap starting.
I20260620 18:47:29.169345 28529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 185079f219cb40858b2244200f74337e: Neither blocks nor log segments found. Creating new log.
I20260620 18:47:29.170711 28529 log.cc:824] T 00000000000000000000000000000000 P 185079f219cb40858b2244200f74337e: Log is configured to *not* fsync() on all Append() calls
I20260620 18:47:29.175778 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:29.086347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60378 (local address 127.5.225.189:40581)
0620 18:47:29.086589 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:29.086597 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:29.112613 (+ 26016us) server_negotiation.cc:407] Connection header received
0620 18:47:29.116246 (+ 3633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:29.116252 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:29.116334 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:29.116411 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:29.120590 (+ 4179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:29.121861 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:29.125252 (+ 3391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:29.125950 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:29.126125 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:29.134530 (+ 8405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:29.134540 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:29.134547 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:29.134606 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:29.143088 (+ 8482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:29.149797 (+ 6709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:29.149804 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:29.149807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:29.149991 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:29.172604 (+ 22613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:29.172611 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:29.172615 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:29.173049 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:29.173314 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:29.174861 (+ 1547us) server_negotiation.cc:299] Negotiation successful
0620 18:47:29.175226 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 18:47:29.176808 28529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 185079f219cb40858b2244200f74337e: No bootstrap required, opened a new log
I20260620 18:47:29.178110 28611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cd8338d53444eed8afe04c94aec0203 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d2a742026ac4e35b5221e632a8e08ba. Latest consensus state: current_term: 1 leader_uuid: "8d2a742026ac4e35b5221e632a8e08ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2a742026ac4e35b5221e632a8e08ba" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39507 } } peers { permanent_uuid: "185079f219cb40858b2244200f74337e" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 40581 } } peers { permanent_uuid: "2cd8338d53444eed8afe04c94aec0203" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40777 } } }
I20260620 18:47:29.178489 28611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cd8338d53444eed8afe04c94aec0203 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:47:29.193061 28529 raft_consensus.cc:348] T 00000000000000000000000000000000 P 185079f219cb40858b2244200f74337e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2a742026ac4e35b5221e632a8e08ba" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39507 } } peers { permanent_uuid: "185079f219cb40858b2244200f74337e" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 40581 } } peers { permanent_uuid: "2cd8338d53444eed8afe04c94aec0203" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40777 } }
I20260620 18:47:29.194105 28529 raft_consensus.cc:729] T 00000000000000000000000000000000 P 185079f219cb40858b2244200f74337e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 185079f219cb40858b2244200f74337e, State: Initialized, Role: FOLLOWER
I20260620 18:47:29.194947 28529 consensus_queue.cc:260] T 00000000000000000000000000000000 P 185079f219cb40858b2244200f74337e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2a742026ac4e35b5221e632a8e08ba" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39507 } } peers { permanent_uuid: "185079f219cb40858b2244200f74337e" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 40581 } } peers { permanent_uuid: "2cd8338d53444eed8afe04c94aec0203" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40777 } }
I20260620 18:47:29.198683 28611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cd8338d53444eed8afe04c94aec0203 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8d2a742026ac4e35b5221e632a8e08ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2a742026ac4e35b5221e632a8e08ba" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39507 } } peers { permanent_uuid: "185079f219cb40858b2244200f74337e" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 40581 } } peers { permanent_uuid: "2cd8338d53444eed8afe04c94aec0203" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40777 } } }
I20260620 18:47:29.198913 28611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cd8338d53444eed8afe04c94aec0203 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:47:29.199958 28631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d2a742026ac4e35b5221e632a8e08ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8d2a742026ac4e35b5221e632a8e08ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2a742026ac4e35b5221e632a8e08ba" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39507 } } peers { permanent_uuid: "185079f219cb40858b2244200f74337e" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 40581 } } peers { permanent_uuid: "2cd8338d53444eed8afe04c94aec0203" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40777 } } }
I20260620 18:47:29.200304 28631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d2a742026ac4e35b5221e632a8e08ba [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:29.204527 28636 catalog_manager.cc:1347] Generated new cluster ID: 50d2d828a3fe463abcd5d0aecbd1919e
I20260620 18:47:29.204629 28636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:47:29.206651 28655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 185079f219cb40858b2244200f74337e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2a742026ac4e35b5221e632a8e08ba" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39507 } } peers { permanent_uuid: "185079f219cb40858b2244200f74337e" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 40581 } } peers { permanent_uuid: "2cd8338d53444eed8afe04c94aec0203" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40777 } } }
I20260620 18:47:29.207006 28655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 185079f219cb40858b2244200f74337e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:47:29.223989 28529 sys_catalog.cc:564] T 00000000000000000000000000000000 P 185079f219cb40858b2244200f74337e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:47:29.224445 28503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 185079f219cb40858b2244200f74337e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:47:29.227663 28504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d2a742026ac4e35b5221e632a8e08ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "185079f219cb40858b2244200f74337e"
I20260620 18:47:29.233300 28503 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 185079f219cb40858b2244200f74337e [term 1 FOLLOWER]: Refusing update from remote peer 8d2a742026ac4e35b5221e632a8e08ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:47:29.237007 28504 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 185079f219cb40858b2244200f74337e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d2a742026ac4e35b5221e632a8e08ba in term 1.
I20260620 18:47:29.241514 28625 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8d2a742026ac4e35b5221e632a8e08ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "185079f219cb40858b2244200f74337e" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:47:29.246717 28636 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:47:29.248641 28636 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:47:29.296329 28636 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d2a742026ac4e35b5221e632a8e08ba: Generated new TSK 0
I20260620 18:47:29.297502 28636 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:47:29.319500 28663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 185079f219cb40858b2244200f74337e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8d2a742026ac4e35b5221e632a8e08ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2a742026ac4e35b5221e632a8e08ba" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39507 } } peers { permanent_uuid: "185079f219cb40858b2244200f74337e" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 40581 } } peers { permanent_uuid: "2cd8338d53444eed8afe04c94aec0203" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40777 } } }
I20260620 18:47:29.319950 28663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 185079f219cb40858b2244200f74337e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:47:29.320466 28655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 185079f219cb40858b2244200f74337e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d2a742026ac4e35b5221e632a8e08ba. Latest consensus state: current_term: 1 leader_uuid: "8d2a742026ac4e35b5221e632a8e08ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2a742026ac4e35b5221e632a8e08ba" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39507 } } peers { permanent_uuid: "185079f219cb40858b2244200f74337e" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 40581 } } peers { permanent_uuid: "2cd8338d53444eed8afe04c94aec0203" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40777 } } }
I20260620 18:47:29.320753 28655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 185079f219cb40858b2244200f74337e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:47:29.321900 28658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d2a742026ac4e35b5221e632a8e08ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8d2a742026ac4e35b5221e632a8e08ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2a742026ac4e35b5221e632a8e08ba" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 39507 } } peers { permanent_uuid: "185079f219cb40858b2244200f74337e" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 40581 } } peers { permanent_uuid: "2cd8338d53444eed8afe04c94aec0203" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 40777 } } }
I20260620 18:47:29.322366 28658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d2a742026ac4e35b5221e632a8e08ba [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:29.370261 28674 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260620 18:47:29.445749 6022 mini_kdc.cc:348] Time spent kdestroy: real 0.131s user 0.000s sys 0.026s
W20260620 18:47:29.489060 28609 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:47:29.472965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49988 (local address 127.5.225.190:39507)
0620 18:47:29.473208 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:29.473222 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:29.486734 (+ 13512us) server_negotiation.cc:407] Connection header received
0620 18:47:29.487062 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:29.487070 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:29.487165 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:29.487255 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:29.488712 (+ 1457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49988: BlockingRecv error: recv got EOF from 127.0.0.1:49988 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260620 18:47:29.494968 28607 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:47:29.487109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59284 (local address 127.5.225.188:40777)
0620 18:47:29.487305 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:29.487311 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:29.492336 (+ 5025us) server_negotiation.cc:407] Connection header received
0620 18:47:29.493237 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:29.493244 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:29.493342 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:29.493421 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:29.494650 (+ 1229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59284: BlockingRecv error: recv got EOF from 127.0.0.1:59284 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 18:47:29.500898 28534 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:47:29.479736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60386 (local address 127.5.225.189:40581)
0620 18:47:29.479951 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:29.479961 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:29.489775 (+ 9814us) server_negotiation.cc:407] Connection header received
0620 18:47:29.499304 (+ 9529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:29.499311 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:29.499410 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:29.499516 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:29.500651 (+ 1135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60386: BlockingRecv error: recv got EOF from 127.0.0.1:60386 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 18:47:29.543943 28534 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:47:29.536655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60396 (local address 127.5.225.189:40581)
0620 18:47:29.536873 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:29.536880 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:29.541858 (+ 4978us) server_negotiation.cc:407] Connection header received
0620 18:47:29.542148 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:29.542155 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:29.542334 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:29.542452 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:29.543643 (+ 1191us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60396: BlockingRecv error: recv got EOF from 127.0.0.1:60396 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260620 18:47:29.583554 28534 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:47:29.573643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60402 (local address 127.5.225.189:40581)
0620 18:47:29.573848 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:29.573855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:29.581435 (+ 7580us) server_negotiation.cc:407] Connection header received
0620 18:47:29.582007 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:29.582015 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:29.582118 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:29.582226 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:29.583315 (+ 1089us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60402: BlockingRecv error: recv got EOF from 127.0.0.1:60402 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 18:47:29.639911 28534 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:47:29.628642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60410 (local address 127.5.225.189:40581)
0620 18:47:29.628846 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:29.628853 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:29.636632 (+ 7779us) server_negotiation.cc:407] Connection header received
0620 18:47:29.636886 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:29.636893 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:29.637038 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:29.637148 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:29.639617 (+ 2469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60410: BlockingRecv error: recv got EOF from 127.0.0.1:60410 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 18:47:29.700827 28534 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:47:29.692204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60424 (local address 127.5.225.189:40581)
0620 18:47:29.692455 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:29.692462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:29.698846 (+ 6384us) server_negotiation.cc:407] Connection header received
0620 18:47:29.699081 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:29.699087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:29.699193 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:29.699340 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:29.700524 (+ 1184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60424: BlockingRecv error: recv got EOF from 127.0.0.1:60424 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
W20260620 18:47:29.745019 28534 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:47:29.743107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60434 (local address 127.5.225.189:40581)
0620 18:47:29.743290 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:29.743295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:29.743682 (+ 387us) server_negotiation.cc:407] Connection header received
0620 18:47:29.743840 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:29.743846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:29.743952 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:29.744083 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:29.744781 (+ 698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60434: BlockingRecv error: recv got EOF from 127.0.0.1:60434 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 18:47:29.766497 28534 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:47:29.764331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60438 (local address 127.5.225.189:40581)
0620 18:47:29.764539 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:29.764548 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:29.764883 (+ 335us) server_negotiation.cc:407] Connection header received
0620 18:47:29.765121 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:29.765125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:29.765216 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:29.765339 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:29.766145 (+ 806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60438: BlockingRecv error: recv got EOF from 127.0.0.1:60438 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 18:47:29.804721 28534 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:47:29.802489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60446 (local address 127.5.225.189:40581)
0620 18:47:29.802727 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:29.802732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:29.803173 (+ 441us) server_negotiation.cc:407] Connection header received
0620 18:47:29.803337 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:29.803342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:29.803416 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:29.803542 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:29.804457 (+ 915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60446: BlockingRecv error: recv got EOF from 127.0.0.1:60446 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
W20260620 18:47:29.874845 28534 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:47:29.872854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60448 (local address 127.5.225.189:40581)
0620 18:47:29.873073 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:29.873080 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:29.873454 (+ 374us) server_negotiation.cc:407] Connection header received
0620 18:47:29.873629 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:29.873636 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:29.873745 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:29.873871 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:29.874629 (+ 758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60448: BlockingRecv error: recv got EOF from 127.0.0.1:60448 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 18:47:30.008627 28534 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:47:30.006655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60462 (local address 127.5.225.189:40581)
0620 18:47:30.006902 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:30.006908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:30.007238 (+ 330us) server_negotiation.cc:407] Connection header received
0620 18:47:30.007397 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:30.007402 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:30.007477 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:30.007570 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:30.008336 (+ 766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60462: BlockingRecv error: recv got EOF from 127.0.0.1:60462 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 18:47:30.027175 28622 catalog_manager.cc:1259] Loaded cluster ID: 50d2d828a3fe463abcd5d0aecbd1919e
I20260620 18:47:30.027280 28622 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2cd8338d53444eed8afe04c94aec0203: loading cluster ID for follower catalog manager: success
I20260620 18:47:30.029778 28622 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2cd8338d53444eed8afe04c94aec0203: acquiring CA information for follower catalog manager: success
I20260620 18:47:30.031699 28622 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2cd8338d53444eed8afe04c94aec0203: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 18:47:30.270536 28534 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:47:30.268537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60478 (local address 127.5.225.189:40581)
0620 18:47:30.268794 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:30.268800 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:30.269112 (+ 312us) server_negotiation.cc:407] Connection header received
0620 18:47:30.269375 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:30.269379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:30.269450 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:30.269558 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:30.270372 (+ 814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60478: BlockingRecv error: recv got EOF from 127.0.0.1:60478 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:47:30.374588 28674 catalog_manager.cc:1259] Loaded cluster ID: 50d2d828a3fe463abcd5d0aecbd1919e
I20260620 18:47:30.374706 28674 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 185079f219cb40858b2244200f74337e: loading cluster ID for follower catalog manager: success
I20260620 18:47:30.377480 28674 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 185079f219cb40858b2244200f74337e: acquiring CA information for follower catalog manager: success
I20260620 18:47:30.379284 28674 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 185079f219cb40858b2244200f74337e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:47:30.793668 28739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:30.787810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60480 (local address 127.5.225.189:40581)
0620 18:47:30.788357 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:30.788362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:30.788455 (+ 93us) server_negotiation.cc:407] Connection header received
0620 18:47:30.788621 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:30.788625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:30.788786 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:30.788904 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:47:30.789352 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:30.790129 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:30.791478 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:30.791912 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:30.792010 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:30.792456 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:47:30.793022 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:47:30.793134 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:30.793237 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 18:47:30.793306 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":311,"threads_started":1}
I20260620 18:47:30.806329 28745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:30.800345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59292 (local address 127.5.225.188:40777)
0620 18:47:30.800831 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:30.800836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:30.800948 (+ 112us) server_negotiation.cc:407] Connection header received
0620 18:47:30.801130 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:30.801135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:30.801278 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:30.801419 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:47:30.801855 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:30.802680 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:30.804208 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:30.804632 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:30.804776 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:30.805200 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:47:30.805747 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:47:30.805861 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:30.805938 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 18:47:30.806004 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":241,"threads_started":1}
I20260620 18:47:30.824520 28751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:30.812462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49994 (local address 127.5.225.190:39507)
0620 18:47:30.816272 (+ 3810us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:30.816282 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:30.817021 (+ 739us) server_negotiation.cc:407] Connection header received
0620 18:47:30.817343 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:30.817352 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:30.817658 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:30.817878 (+ 220us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:47:30.818584 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:30.819835 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:30.822031 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:30.822628 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:30.822763 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:30.823238 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:47:30.823565 (+ 327us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:47:30.823690 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:30.824077 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 18:47:30.824164 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3642,"thread_start_us":3507,"threads_started":1}
I20260620 18:47:30.826365 28745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:30.812712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59306 (local address 127.5.225.188:40777)
0620 18:47:30.816234 (+ 3522us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:30.816242 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:30.819882 (+ 3640us) server_negotiation.cc:407] Connection header received
0620 18:47:30.819958 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:30.819964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:30.820049 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:30.820156 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:47:30.820812 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:30.821521 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:30.823787 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:30.824406 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:30.824570 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:30.825140 (+ 570us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:47:30.825261 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:47:30.825339 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:30.825583 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 18:47:30.825668 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3375}
I20260620 18:47:30.826689 28739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:30.812848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60494 (local address 127.5.225.189:40581)
0620 18:47:30.814915 (+ 2067us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:30.814925 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:30.819441 (+ 4516us) server_negotiation.cc:407] Connection header received
0620 18:47:30.819626 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:30.819633 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:30.819725 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:30.819846 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:47:30.820534 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:30.821633 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:30.824675 (+ 3042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:30.825267 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:30.825406 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:30.825895 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:47:30.826002 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:47:30.826102 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:30.826274 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 18:47:30.826361 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1886}
I20260620 18:47:30.830566 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 28388
I20260620 18:47:30.831691 28450 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:47:30.943526 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 28388
W20260620 18:47:30.961328 28480 connection.cc:535] server connection from 127.0.0.1:60340 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 18:47:30.961902 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 28462
I20260620 18:47:30.963140 28524 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:47:31.070483 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 28462
I20260620 18:47:31.090920 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 28537
I20260620 18:47:31.092048 28599 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:47:31.200417 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 28537
2026-06-20T18:47:31Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5191 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:47:31 dist-test-slave-k6lg krb5kdc[28759](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:47:31 dist-test-slave-k6lg krb5kdc[28759](info): set up 2 sockets
Jun 20 18:47:31 dist-test-slave-k6lg krb5kdc[28759](info): commencing operation
krb5kdc: starting...
W20260620 18:47:32.334975 6022 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/krb5kdc/test-user.keytab.
Jun 20 18:47:32 dist-test-slave-k6lg krb5kdc[28759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:47:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:47:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
Entry for principal kudu/127.5.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab.
I20260620 18:47:32.593245 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:43813
--webserver_interface=127.5.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:44403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:43813,127.5.225.189:39353,127.5.225.188:39547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:47:32.919409 28772 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:47:32.919739 28772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:32.919845 28772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:32.919999 28772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:32.927188 28772 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:47:32.927270 28772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:32.927366 28772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:32.927452 28772 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:47:32.927565 28772 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:47:32.937225 28772 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:44403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:43813,127.5.225.189:39353,127.5.225.188:39547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.190.keytab
--principal=kudu/127.5.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.190:43813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:47:32.938632 28772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:32.939973 28772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:32.951843 28777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:32.951939 28778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:32.951965 28780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:32.952657 28772 server_base.cc:1034] running on GCE node
I20260620 18:47:32.953336 28772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:32.954759 28772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:32.955976 28772 hybrid_clock.cc:648] HybridClock initialized: now 1781981252955917 us; error 28 us; skew 500 ppm
I20260620 18:47:32.956254 28772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:32 dist-test-slave-k6lg krb5kdc[28759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981252, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:32.962149 28772 init.cc:377] Logged in from keytab as kudu/127.5.225.190@KRBTEST.COM (short username kudu)
I20260620 18:47:32.963588 28772 webserver.cc:460] Webserver started at http://127.5.225.190:38467/ using document root <none> and password file <none>
I20260620 18:47:32.964262 28772 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:32.964350 28772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:32.964622 28772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:47:32.967677 28772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-0/data/instance:
uuid: "517e56825ff4444ea601c998fd470ac1"
format_stamp: "Formatted at 2026-06-20 18:47:32 on dist-test-slave-k6lg"
server_key: "0595164c840e6722db69982a6f73b031"
server_key_iv: "7c3791b03e963518779ff6d2ec75e7b4"
server_key_version: "encryptionkey@0"
I20260620 18:47:32.968379 28772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-0/wal/instance:
uuid: "517e56825ff4444ea601c998fd470ac1"
format_stamp: "Formatted at 2026-06-20 18:47:32 on dist-test-slave-k6lg"
server_key: "0595164c840e6722db69982a6f73b031"
server_key_iv: "7c3791b03e963518779ff6d2ec75e7b4"
server_key_version: "encryptionkey@0"
I20260620 18:47:32.973384 28772 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:47:32.976785 28787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:32.978112 28772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 18:47:32.978330 28772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-0/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-0/wal
uuid: "517e56825ff4444ea601c998fd470ac1"
format_stamp: "Formatted at 2026-06-20 18:47:32 on dist-test-slave-k6lg"
server_key: "0595164c840e6722db69982a6f73b031"
server_key_iv: "7c3791b03e963518779ff6d2ec75e7b4"
server_key_version: "encryptionkey@0"
I20260620 18:47:32.978521 28772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:32.993846 28772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:32.998299 28772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:32.998525 28772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:33.016713 28772 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.190:43813
I20260620 18:47:33.016742 28838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.190:43813 every 8 connection(s)
I20260620 18:47:33.018085 28772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-0/data/info.pb
I20260620 18:47:33.021997 28839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:47:33.026696 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 28772
I20260620 18:47:33.026889 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-0/wal/instance
I20260620 18:47:33.027351 6022 external_mini_cluster.cc:1428] Setting key 2fbf3c66ae244d08f143b20045599a1b
I20260620 18:47:33.028759 28839 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.225.190" port: 43813 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:47:33 dist-test-slave-k6lg krb5kdc[28759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981252, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.225.189@KRBTEST.COM; defaulting to no policy
I20260620 18:47:33.052384 28842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:33.033439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37674 (local address 127.5.225.190:43813)
0620 18:47:33.034259 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:33.034268 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:33.034327 (+ 59us) server_negotiation.cc:407] Connection header received
0620 18:47:33.035803 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:33.035809 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:33.036041 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:33.036247 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:33.037527 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:33.038776 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:33.040112 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:33.040653 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:33.040847 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:33.044973 (+ 4126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:33.044984 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:33.045000 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:33.045037 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:33.047933 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:33.048895 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:33.048902 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:33.048904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:33.049067 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:33.049547 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:33.049550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:33.049552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:33.050053 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:33.050293 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:33.050896 (+ 603us) server_negotiation.cc:299] Negotiation successful
0620 18:47:33.051192 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":589,"thread_start_us":434,"threads_started":1}
I20260620 18:47:33.052431 28841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:33.032121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:43813 (local address 127.0.0.1:37674)
0620 18:47:33.032918 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:33.032957 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:33.035420 (+ 2463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:33.036252 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:33.036258 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:33.036985 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:47:33.037310 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:33.037326 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:33.038923 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:33.038928 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:33.039959 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:33.039986 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:33.040874 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:33.040880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:33.041093 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:33.041776 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:47:33.041801 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:47:33.044717 (+ 2916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:47:33.048081 (+ 3364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:33.048096 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:33.048110 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:33.048743 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:33.049186 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:33.049194 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:33.049201 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:33.049428 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:33.050225 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:47:33.050235 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:47:33.050492 (+ 257us) client_negotiation.cc:769] Sending connection context
0620 18:47:33.050793 (+ 301us) client_negotiation.cc:240] Negotiation successful
0620 18:47:33.051385 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":86,"thread_start_us":332,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.189@KRBTEST.COM" created.
I20260620 18:47:33.055681 28839 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.225.189" port: 39353 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:47:33.061987 28841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:33.056485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:39353 (local address 127.0.0.1:39720)
0620 18:47:33.061710 (+ 5225us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:33.061846 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.189:39353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5090}
W20260620 18:47:33.062466 28789 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.225.189:39353: connect: Connection refused (error 111)
I20260620 18:47:33.065716 28845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:33.064600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:39353 (local address 127.0.0.1:39732)
0620 18:47:33.065239 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:33.065393 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.189:39353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":270,"threads_started":1}
W20260620 18:47:33.065943 28839 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.225.189:39353: Network error: Client connection negotiation failed: client connection to 127.5.225.189:39353: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.189.keytab.
Entry for principal kudu/127.5.225.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.189.keytab.
I20260620 18:47:33.087610 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.189:39353
--webserver_interface=127.5.225.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.189.keytab
--principal=kudu/127.5.225.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:44403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:43813,127.5.225.189:39353,127.5.225.188:39547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:47:33.115303 28839 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.225.189" port: 39353 } attempt: 1
I20260620 18:47:33.116549 28845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:33.116085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:39353 (local address 127.0.0.1:39744)
0620 18:47:33.116295 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:33.116394 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.189:39353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 18:47:33.117573 28845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:33.117209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:39353 (local address 127.0.0.1:39752)
0620 18:47:33.117368 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:33.117447 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.189:39353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 18:47:33.117764 28839 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.225.189:39353: Network error: Client connection negotiation failed: client connection to 127.5.225.189:39353: connect: Connection refused (error 111)
I20260620 18:47:33.186030 28839 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.225.189" port: 39353 } attempt: 2
I20260620 18:47:33.187253 28845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:33.186762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:39353 (local address 127.0.0.1:39764)
0620 18:47:33.187007 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:33.187107 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.189:39353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 18:47:33.188345 28845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:33.187958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:39353 (local address 127.0.0.1:39774)
0620 18:47:33.188141 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:33.188218 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.189:39353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260620 18:47:33.188634 28839 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.225.189:39353: Network error: Client connection negotiation failed: client connection to 127.5.225.189:39353: connect: Connection refused (error 111)
I20260620 18:47:33.279907 28839 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.225.189" port: 39353 } attempt: 3
I20260620 18:47:33.281160 28845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:33.280705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:39353 (local address 127.0.0.1:39780)
0620 18:47:33.280928 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:33.281023 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.189:39353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 18:47:33.282274 28845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:33.281832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:39353 (local address 127.0.0.1:39788)
0620 18:47:33.281984 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:33.282068 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.189:39353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 18:47:33.282505 28839 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.225.189:39353: Network error: Client connection negotiation failed: client connection to 127.5.225.189:39353: connect: Connection refused (error 111)
W20260620 18:47:33.416966 28847 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:47:33.417277 28847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:33.417351 28847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:33.417431 28847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:33.424728 28847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:47:33.424821 28847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:33.424878 28847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:33.424921 28847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:47:33.424985 28847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:47:33.425774 28839 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.225.189" port: 39353 } attempt: 4
I20260620 18:47:33.427057 28845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:33.426441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:39353 (local address 127.0.0.1:39796)
0620 18:47:33.426765 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:33.426889 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.189:39353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260620 18:47:33.428171 28845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:33.427751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:39353 (local address 127.0.0.1:39812)
0620 18:47:33.427925 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:33.428006 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.189:39353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 18:47:33.428390 28839 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.225.189:39353: Network error: Client connection negotiation failed: client connection to 127.5.225.189:39353: connect: Connection refused (error 111)
I20260620 18:47:33.436781 28847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:44403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:43813,127.5.225.189:39353,127.5.225.188:39547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.189.keytab
--principal=kudu/127.5.225.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.189:39353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.225.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:47:33.438052 28847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:33.439419 28847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:33.450225 28852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:33.451175 28853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:33.452036 28855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:33.452641 28847 server_base.cc:1034] running on GCE node
I20260620 18:47:33.453438 28847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:33.454836 28847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:33.456054 28847 hybrid_clock.cc:648] HybridClock initialized: now 1781981253455988 us; error 37 us; skew 500 ppm
I20260620 18:47:33.456370 28847 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:33 dist-test-slave-k6lg krb5kdc[28759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981253, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:33.459643 28847 init.cc:377] Logged in from keytab as kudu/127.5.225.189@KRBTEST.COM (short username kudu)
I20260620 18:47:33.461234 28847 webserver.cc:460] Webserver started at http://127.5.225.189:40935/ using document root <none> and password file <none>
I20260620 18:47:33.461939 28847 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:33.462059 28847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:33.462406 28847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:47:33.465258 28847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-1/data/instance:
uuid: "40a4841f366c47de98e1a4b354eec5ff"
format_stamp: "Formatted at 2026-06-20 18:47:33 on dist-test-slave-k6lg"
server_key: "162306542eff4930636f54c1bf63a213"
server_key_iv: "df446bf44ece15ed957a42fe621dac2c"
server_key_version: "encryptionkey@0"
I20260620 18:47:33.466074 28847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-1/wal/instance:
uuid: "40a4841f366c47de98e1a4b354eec5ff"
format_stamp: "Formatted at 2026-06-20 18:47:33 on dist-test-slave-k6lg"
server_key: "162306542eff4930636f54c1bf63a213"
server_key_iv: "df446bf44ece15ed957a42fe621dac2c"
server_key_version: "encryptionkey@0"
I20260620 18:47:33.471184 28847 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:47:33.474604 28862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:33.475901 28847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:47:33.476086 28847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-1/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-1/wal
uuid: "40a4841f366c47de98e1a4b354eec5ff"
format_stamp: "Formatted at 2026-06-20 18:47:33 on dist-test-slave-k6lg"
server_key: "162306542eff4930636f54c1bf63a213"
server_key_iv: "df446bf44ece15ed957a42fe621dac2c"
server_key_version: "encryptionkey@0"
I20260620 18:47:33.476298 28847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:33.492664 28847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:33.496621 28847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:33.496886 28847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:33.515273 28847 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.189:39353
I20260620 18:47:33.515281 28913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.189:39353 every 8 connection(s)
I20260620 18:47:33.516573 28847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-1/data/info.pb
I20260620 18:47:33.520097 28914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:47:33.521884 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 28847
I20260620 18:47:33.522080 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-1/wal/instance
I20260620 18:47:33.522662 6022 external_mini_cluster.cc:1428] Setting key 3c092c7e04d5631a49457eeb95498839
I20260620 18:47:33.529520 28914 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.225.190" port: 43813 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:47:33 dist-test-slave-k6lg krb5kdc[28759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981253, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.189@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.225.188@KRBTEST.COM; defaulting to no policy
I20260620 18:47:33.551429 28842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:33.533129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37678 (local address 127.5.225.190:43813)
0620 18:47:33.533351 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:33.533356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:33.534723 (+ 1367us) server_negotiation.cc:407] Connection header received
0620 18:47:33.536658 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:33.536664 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:33.536762 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:33.536876 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:33.538465 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:33.539190 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:33.540742 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:33.541183 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:33.541321 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:33.545192 (+ 3871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:33.545197 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:33.545200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:33.545233 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:33.548253 (+ 3020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:33.549104 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:33.549108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:33.549111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:33.549232 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:33.549686 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:33.549690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:33.549692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:33.549974 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:33.550087 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:33.550986 (+ 899us) server_negotiation.cc:299] Negotiation successful
0620 18:47:33.551156 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.225.188@KRBTEST.COM" created.
I20260620 18:47:33.552495 28917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:33.533663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:43813 (local address 127.0.0.1:37678)
0620 18:47:33.534548 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:33.534586 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:33.536374 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:33.537120 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:33.537128 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:33.537660 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:47:33.538290 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:33.538309 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:33.539359 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:33.539365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:33.540568 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:33.540580 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:33.541410 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:33.541417 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:33.541655 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:33.542379 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:47:33.542406 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:47:33.544933 (+ 2527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:47:33.548401 (+ 3468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:33.548415 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:33.548437 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:33.548992 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:33.549327 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:33.549330 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:33.549332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:33.549574 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:33.550133 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:47:33.550139 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:47:33.550603 (+ 464us) client_negotiation.cc:769] Sending connection context
0620 18:47:33.550874 (+ 271us) client_negotiation.cc:240] Negotiation successful
0620 18:47:33.551258 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":406,"threads_started":1}
I20260620 18:47:33.554251 28914 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.225.189" port: 39353 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:47:33 dist-test-slave-k6lg krb5kdc[28759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981253, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.189@KRBTEST.COM for kudu/127.5.225.189@KRBTEST.COM
I20260620 18:47:33.572885 28917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:33.555150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:39353 (local address 127.0.0.1:39824)
0620 18:47:33.555366 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:33.555386 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:33.555483 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:33.559237 (+ 3754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:33.559240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:33.559746 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:47:33.559950 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:33.559965 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:33.561341 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:33.561349 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:33.562963 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:33.562981 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:33.563790 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:33.563799 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:33.564026 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:33.564652 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:47:33.564680 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:47:33.567213 (+ 2533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:47:33.570452 (+ 3239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:33.570458 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:33.570462 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:33.570822 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:33.571212 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:33.571217 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:33.571221 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:33.571341 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:33.571921 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:47:33.571927 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:47:33.572067 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 18:47:33.572218 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 18:47:33.572500 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 18:47:33.573007 28918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:33.555145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39824 (local address 127.5.225.189:39353)
0620 18:47:33.558691 (+ 3546us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:33.558701 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:33.558734 (+ 33us) server_negotiation.cc:407] Connection header received
0620 18:47:33.558794 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:33.558804 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:33.559053 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:33.559222 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:33.560128 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:33.561191 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:33.563107 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:33.563632 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:33.563730 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:33.567416 (+ 3686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:33.567426 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:33.567439 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:33.567470 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:33.570314 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:33.570949 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:33.570957 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:33.570960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:33.571078 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:33.571448 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:33.571456 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:33.571459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:33.571803 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:33.572011 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:33.572362 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 18:47:33.572620 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":3352,"thread_start_us":3195,"threads_started":1}
I20260620 18:47:33.575785 28914 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.225.188" port: 39547 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:47:33.576897 28917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:33.576386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:39547 (local address 127.0.0.1:46938)
0620 18:47:33.576560 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:33.576740 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:39547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260620 18:47:33.577131 28864 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.225.188:39547: connect: Connection refused (error 111)
I20260620 18:47:33.578840 28917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:33.578435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:39547 (local address 127.0.0.1:46944)
0620 18:47:33.578572 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:33.578721 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:39547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 18:47:33.579092 28914 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.225.188:39547: Network error: Client connection negotiation failed: client connection to 127.5.225.188:39547: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.225.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.188.keytab.
Entry for principal kudu/127.5.225.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.188.keytab.
I20260620 18:47:33.585260 6022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuQUw40/build/asan/bin/kudu
/tmp/dist-test-taskuQUw40/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.188:39547
--webserver_interface=127.5.225.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.188.keytab
--principal=kudu/127.5.225.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.225.148:44403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.225.190:43813,127.5.225.189:39353,127.5.225.188:39547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:47:33.628399 28914 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.225.188" port: 39547 } attempt: 1
I20260620 18:47:33.629578 28917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:33.629134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:39547 (local address 127.0.0.1:46958)
0620 18:47:33.629327 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:33.629433 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:39547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 18:47:33.630661 28917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:33.630314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:39547 (local address 127.0.0.1:46966)
0620 18:47:33.630476 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:33.630549 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:39547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 18:47:33.630844 28914 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.225.188:39547: Network error: Client connection negotiation failed: client connection to 127.5.225.188:39547: connect: Connection refused (error 111)
I20260620 18:47:33.699075 28914 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.225.188" port: 39547 } attempt: 2
I20260620 18:47:33.700294 28917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:33.699744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:39547 (local address 127.0.0.1:46978)
0620 18:47:33.699979 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:33.700099 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:39547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 18:47:33.701323 28917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:33.700958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:39547 (local address 127.0.0.1:46988)
0620 18:47:33.701106 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:33.701189 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:39547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 18:47:33.701543 28914 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.225.188:39547: Network error: Client connection negotiation failed: client connection to 127.5.225.188:39547: connect: Connection refused (error 111)
I20260620 18:47:33.727620 28839 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.225.189" port: 39353 } attempt: 5
Jun 20 18:47:33 dist-test-slave-k6lg krb5kdc[28759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981252, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for kudu/127.5.225.189@KRBTEST.COM
I20260620 18:47:33.740911 28918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:33.728617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39832 (local address 127.5.225.189:39353)
0620 18:47:33.728812 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:33.728817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:33.728839 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:47:33.728932 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:33.728936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:33.728995 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:33.729103 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:33.729874 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:33.730662 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:33.731846 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:33.732295 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:33.732431 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:33.735822 (+ 3391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:33.735827 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:33.735830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:33.735861 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:33.738424 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:33.739043 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:33.739049 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:33.739053 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:33.739185 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:33.739555 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:33.739561 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:33.739565 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:33.739842 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:33.739996 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:33.740383 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 18:47:33.740589 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 18:47:33.740911 28845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:33.728404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:39353 (local address 127.0.0.1:39832)
0620 18:47:33.728596 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:33.728620 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:33.728794 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:33.729134 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:33.729139 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:33.729550 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:47:33.729718 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:33.729726 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:33.730813 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:33.730818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:33.731688 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:33.731699 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:33.732455 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:33.732461 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:33.732646 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:33.733318 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:47:33.733336 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:47:33.735616 (+ 2280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:47:33.738561 (+ 2945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:33.738566 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:33.738570 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:33.738924 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:33.739297 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:33.739302 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:33.739306 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:33.739447 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:33.739962 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:47:33.739966 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:47:33.740130 (+ 164us) client_negotiation.cc:769] Sending connection context
0620 18:47:33.740273 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 18:47:33.740553 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 18:47:33.742244 28839 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.225.188" port: 39547 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:47:33.743211 28845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:33.742819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:39547 (local address 127.0.0.1:46994)
0620 18:47:33.742994 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:33.743074 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:39547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 18:47:33.744093 28845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:33.743773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:39547 (local address 127.0.0.1:47006)
0620 18:47:33.743931 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:33.744001 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:39547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 18:47:33.744320 28839 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.225.188:39547: Network error: Client connection negotiation failed: client connection to 127.5.225.188:39547: connect: Connection refused (error 111)
I20260620 18:47:33.792855 28914 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.225.188" port: 39547 } attempt: 3
I20260620 18:47:33.794284 28917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:33.793551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:39547 (local address 127.0.0.1:47022)
0620 18:47:33.793859 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:33.794004 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:39547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 18:47:33.795511 28917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:33.795011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:39547 (local address 127.0.0.1:47036)
0620 18:47:33.795255 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:33.795345 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:39547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260620 18:47:33.795579 28839 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.225.188" port: 39547 } attempt: 1
W20260620 18:47:33.795723 28914 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.225.188:39547: Network error: Client connection negotiation failed: client connection to 127.5.225.188:39547: connect: Connection refused (error 111)
I20260620 18:47:33.796541 28845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:33.796166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:39547 (local address 127.0.0.1:47052)
0620 18:47:33.796326 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:33.796406 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:39547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 18:47:33.797580 28845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:33.797276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:39547 (local address 127.0.0.1:47066)
0620 18:47:33.797427 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:33.797488 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:39547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 18:47:33.797816 28839 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.225.188:39547: Network error: Client connection negotiation failed: client connection to 127.5.225.188:39547: connect: Connection refused (error 111)
I20260620 18:47:33.866143 28839 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.225.188" port: 39547 } attempt: 2
I20260620 18:47:33.867451 28845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:33.866861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:39547 (local address 127.0.0.1:47068)
0620 18:47:33.867113 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:33.867227 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:39547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 18:47:33.868626 28845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:33.868156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:39547 (local address 127.0.0.1:47082)
0620 18:47:33.868368 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:33.868469 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:39547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 18:47:33.868892 28839 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.225.188:39547: Network error: Client connection negotiation failed: client connection to 127.5.225.188:39547: connect: Connection refused (error 111)
W20260620 18:47:33.910594 28921 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:47:33.910950 28921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:33.911072 28921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:33.911219 28921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:33.918557 28921 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:47:33.918648 28921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:33.918773 28921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:33.918843 28921 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:47:33.918927 28921 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:47:33.928925 28921 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.225.148:44403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.225.190:43813,127.5.225.189:39353,127.5.225.188:39547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/krb5kdc/kudu_127.5.225.188.keytab
--principal=kudu/127.5.225.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.225.188:39547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.225.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:47:33.930315 28921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:33.931663 28921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 18:47:33.938949 28914 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.225.188" port: 39547 } attempt: 4
I20260620 18:47:33.940212 28917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:33.939714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:39547 (local address 127.0.0.1:47084)
0620 18:47:33.939956 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:33.940059 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:39547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 18:47:33.941226 28917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:33.940917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:39547 (local address 127.0.0.1:47086)
0620 18:47:33.941056 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:33.941130 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:39547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 18:47:33.941476 28914 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.225.188:39547: Network error: Client connection negotiation failed: client connection to 127.5.225.188:39547: connect: Connection refused (error 111)
W20260620 18:47:33.943456 28927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:33.943430 28926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:33.944080 28929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:33.944562 28921 server_base.cc:1034] running on GCE node
I20260620 18:47:33.945420 28921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:33.946949 28921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:33.948166 28921 hybrid_clock.cc:648] HybridClock initialized: now 1781981253948099 us; error 41 us; skew 500 ppm
I20260620 18:47:33.948516 28921 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:33 dist-test-slave-k6lg krb5kdc[28759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981253, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:33.951824 28921 init.cc:377] Logged in from keytab as kudu/127.5.225.188@KRBTEST.COM (short username kudu)
I20260620 18:47:33.953320 28921 webserver.cc:460] Webserver started at http://127.5.225.188:41673/ using document root <none> and password file <none>
I20260620 18:47:33.954007 28921 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:33.954120 28921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:33.954466 28921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:47:33.957296 28921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-2/data/instance:
uuid: "39f40e16a389406b909677bdc1975dc9"
format_stamp: "Formatted at 2026-06-20 18:47:33 on dist-test-slave-k6lg"
server_key: "d79c8e89682133ffba8604742cfeab99"
server_key_iv: "ee5434e848a2c2f821c01a7e5d9abe50"
server_key_version: "encryptionkey@0"
I20260620 18:47:33.958022 28921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-2/wal/instance:
uuid: "39f40e16a389406b909677bdc1975dc9"
format_stamp: "Formatted at 2026-06-20 18:47:33 on dist-test-slave-k6lg"
server_key: "d79c8e89682133ffba8604742cfeab99"
server_key_iv: "ee5434e848a2c2f821c01a7e5d9abe50"
server_key_version: "encryptionkey@0"
I20260620 18:47:33.963405 28921 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:47:33.966827 28936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:33.968267 28921 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:47:33.968449 28921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-2/data,/tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-2/wal
uuid: "39f40e16a389406b909677bdc1975dc9"
format_stamp: "Formatted at 2026-06-20 18:47:33 on dist-test-slave-k6lg"
server_key: "d79c8e89682133ffba8604742cfeab99"
server_key_iv: "ee5434e848a2c2f821c01a7e5d9abe50"
server_key_version: "encryptionkey@0"
I20260620 18:47:33.968665 28921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:33.975114 28839 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.225.188" port: 39547 } attempt: 3
I20260620 18:47:33.976190 28845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:33.975707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:39547 (local address 127.0.0.1:47100)
0620 18:47:33.975899 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:33.975996 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:39547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 18:47:33.977113 28845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:33.976802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:39547 (local address 127.0.0.1:47114)
0620 18:47:33.976932 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:33.977013 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.225.188:39547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 18:47:33.977389 28839 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.225.188:39547: Network error: Client connection negotiation failed: client connection to 127.5.225.188:39547: connect: Connection refused (error 111)
I20260620 18:47:33.984426 28921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:33.988679 28921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:33.988945 28921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:34.008059 28921 rpc_server.cc:307] RPC server started. Bound to: 127.5.225.188:39547
I20260620 18:47:34.008148 28987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.225.188:39547 every 8 connection(s)
I20260620 18:47:34.009387 28921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-2/data/info.pb
I20260620 18:47:34.013065 28988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:47:34.019086 6022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuQUw40/build/asan/bin/kudu as pid 28921
I20260620 18:47:34.019287 6022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuQUw40/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980894474392-6022-0/minicluster-data/master-2/wal/instance
I20260620 18:47:34.019753 6022 external_mini_cluster.cc:1428] Setting key fdb6a4a3420b19d590ac2e5e06d481b3
I20260620 18:47:34.021471 28988 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.225.190" port: 43813 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:47:34 dist-test-slave-k6lg krb5kdc[28759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981253, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.188@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
Jun 20 18:47:34 dist-test-slave-k6lg krb5kdc[28759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.188@KRBTEST.COM
I20260620 18:47:34.041697 28842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:34.024185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37688 (local address 127.5.225.190:43813)
0620 18:47:34.024367 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:34.024373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:34.025161 (+ 788us) server_negotiation.cc:407] Connection header received
0620 18:47:34.025614 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:34.025622 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:34.025701 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:34.025808 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:34.027248 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:34.027986 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:34.029871 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:34.030497 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:34.030639 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:34.035440 (+ 4801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:34.035445 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:34.035448 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:34.035481 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:34.038483 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:34.039313 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:34.039319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:34.039324 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:34.039467 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:34.039908 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:34.039915 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:34.039918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:34.040192 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:34.040345 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:34.041089 (+ 744us) server_negotiation.cc:299] Negotiation successful
0620 18:47:34.041314 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 18:47:34.042809 28991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:34.021559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47128 (local address 127.5.225.188:39547)
0620 18:47:34.022440 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:34.022456 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:34.022502 (+ 46us) server_negotiation.cc:407] Connection header received
0620 18:47:34.024995 (+ 2493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:34.025027 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:34.025433 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:34.026292 (+ 859us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:34.027347 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:34.028771 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:34.030427 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:34.031188 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:34.031282 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:34.035942 (+ 4660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:34.035951 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:34.035965 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:34.035998 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:34.039024 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:34.039771 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:34.039777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:34.039779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:34.039903 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:34.040297 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:34.040301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:34.040302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:34.040701 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:34.041044 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:34.041327 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 18:47:34.041668 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":125,"server-negotiator.queue_time_us":633,"thread_start_us":362,"threads_started":1}
I20260620 18:47:34.042847 28992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:34.024182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.190:43813 (local address 127.0.0.1:37688)
0620 18:47:34.025020 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:34.025066 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:34.025379 (+ 313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:34.025874 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:34.025884 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:34.026691 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:47:34.027085 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:34.027102 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:34.028156 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:34.028165 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:34.029695 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:34.029710 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:34.030753 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:34.030759 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:34.031230 (+ 471us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:34.032153 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:47:34.032185 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:47:34.035230 (+ 3045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:47:34.038625 (+ 3395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:34.038643 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:34.038655 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:34.039183 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:34.039585 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:34.039589 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:34.039593 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:34.039794 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:34.040326 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:47:34.040334 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:47:34.040735 (+ 401us) client_negotiation.cc:769] Sending connection context
0620 18:47:34.040996 (+ 261us) client_negotiation.cc:240] Negotiation successful
0620 18:47:34.041532 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":154,"spinlock_wait_cycles":14720,"thread_start_us":328,"threads_started":1}
I20260620 18:47:34.045248 28988 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.225.189" port: 39353 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:47:34 dist-test-slave-k6lg krb5kdc[28759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981253, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.188@KRBTEST.COM for kudu/127.5.225.189@KRBTEST.COM
Jun 20 18:47:34 dist-test-slave-k6lg krb5kdc[28759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.190@KRBTEST.COM
I20260620 18:47:34.058548 28992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:34.046147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.189:39353 (local address 127.0.0.1:39846)
0620 18:47:34.046415 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:34.046434 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:34.046549 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:34.046882 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:34.046887 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:34.047239 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:47:34.047369 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:34.047377 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:34.048306 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:34.048313 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:34.049174 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:34.049185 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:34.049833 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:34.049838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:34.050047 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:34.050657 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:47:34.050680 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:47:34.052769 (+ 2089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:47:34.055961 (+ 3192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:34.055967 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:34.055971 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:34.056493 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:34.056885 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:34.056890 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:34.056893 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:34.057047 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:34.057568 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:47:34.057572 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:47:34.057726 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 18:47:34.057870 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 18:47:34.058158 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168}
I20260620 18:47:34.058583 28918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:34.046263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39846 (local address 127.5.225.189:39353)
0620 18:47:34.046434 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:34.046441 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:34.046482 (+ 41us) server_negotiation.cc:407] Connection header received
0620 18:47:34.046686 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:34.046690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:34.046747 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:34.046851 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:34.047505 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:34.048166 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:34.049306 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:34.049698 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:34.049825 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:34.052958 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:34.052963 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:34.052966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:34.052996 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:34.055829 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:34.056613 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:34.056620 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:34.056623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:34.056761 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:34.057152 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:34.057159 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:34.057162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:34.057442 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:34.057592 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:34.057970 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 18:47:34.058210 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:47:34.059706 28988 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.225.188" port: 39547 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:47:34.060343 28842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:34.046066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37702 (local address 127.5.225.190:43813)
0620 18:47:34.046283 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:34.046288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:34.046598 (+ 310us) server_negotiation.cc:407] Connection header received
0620 18:47:34.046775 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:34.046780 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:34.046845 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:34.046946 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:34.048110 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:34.048837 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:34.050054 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:34.050568 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:34.050699 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:34.054717 (+ 4018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:34.054723 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:34.054726 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:34.054757 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:34.057575 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:34.058291 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:34.058299 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:34.058307 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:34.058435 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:34.058905 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:34.058910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:34.058913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:34.059231 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:34.059509 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:34.059782 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 18:47:34.060011 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 18:47:34 dist-test-slave-k6lg krb5kdc[28759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981253, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.188@KRBTEST.COM for kudu/127.5.225.188@KRBTEST.COM
Jun 20 18:47:34 dist-test-slave-k6lg krb5kdc[28759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.225.189@KRBTEST.COM
I20260620 18:47:34.073854 28992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:34.060415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:39547 (local address 127.0.0.1:47138)
0620 18:47:34.060683 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:34.060700 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:34.060807 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:34.061165 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:34.061170 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:34.061593 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:47:34.061736 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:34.061745 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:34.062948 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:34.062955 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:34.064045 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:34.064061 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:34.064796 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:34.064800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:34.064996 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:34.065545 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:47:34.065561 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:47:34.067904 (+ 2343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:47:34.071300 (+ 3396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:34.071308 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:34.071313 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:34.071796 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:34.072167 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:34.072169 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:34.072180 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:34.072319 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:34.072925 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:47:34.072928 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:47:34.073056 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 18:47:34.073178 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 18:47:34.073452 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":19}
I20260620 18:47:34.074072 28991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:34.060492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47138 (local address 127.5.225.188:39547)
0620 18:47:34.060644 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:34.060649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:34.060770 (+ 121us) server_negotiation.cc:407] Connection header received
0620 18:47:34.060920 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:34.060925 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:34.061002 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:34.061104 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:34.061864 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:34.062833 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:34.064186 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:34.064639 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:34.064788 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:34.068102 (+ 3314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:34.068110 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:34.068117 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:34.068154 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:34.071171 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:34.071905 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:34.071910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:34.071914 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:34.072059 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:34.072443 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:34.072454 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:34.072457 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:34.072815 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:34.072970 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:34.073304 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 18:47:34.073645 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":35}
I20260620 18:47:34.077625 28918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:34.061977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39848 (local address 127.5.225.189:39353)
0620 18:47:34.062133 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:34.062139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:34.062155 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:47:34.062332 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:34.062337 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:34.062405 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:34.062506 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:34.063553 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:34.064468 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:34.065982 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:34.066688 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:34.066812 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:34.071697 (+ 4885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:34.071702 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:34.071705 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:34.071734 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:34.075054 (+ 3320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:34.075661 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:34.075668 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:34.075671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:34.075803 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:34.076157 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:34.076164 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:34.076167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:34.076431 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:34.076689 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:34.076975 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 18:47:34.077202 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:47:34.078804 28988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39f40e16a389406b909677bdc1975dc9: Bootstrap starting.
I20260620 18:47:34.082551 28988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39f40e16a389406b909677bdc1975dc9: Neither blocks nor log segments found. Creating new log.
I20260620 18:47:34.083781 28988 log.cc:824] T 00000000000000000000000000000000 P 39f40e16a389406b909677bdc1975dc9: Log is configured to *not* fsync() on all Append() calls
I20260620 18:47:34.084873 6022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:47:34.086700 28988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39f40e16a389406b909677bdc1975dc9: No bootstrap required, opened a new log
I20260620 18:47:34.087282 28814 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "517e56825ff4444ea601c998fd470ac1"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37702
I20260620 18:47:34.093899 28988 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39f40e16a389406b909677bdc1975dc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517e56825ff4444ea601c998fd470ac1" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 43813 } } peers { permanent_uuid: "40a4841f366c47de98e1a4b354eec5ff" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 39353 } } peers { permanent_uuid: "39f40e16a389406b909677bdc1975dc9" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 39547 } }
I20260620 18:47:34.094280 28988 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39f40e16a389406b909677bdc1975dc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f40e16a389406b909677bdc1975dc9, State: Initialized, Role: FOLLOWER
I20260620 18:47:34.094817 28988 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39f40e16a389406b909677bdc1975dc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517e56825ff4444ea601c998fd470ac1" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 43813 } } peers { permanent_uuid: "40a4841f366c47de98e1a4b354eec5ff" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 39353 } } peers { permanent_uuid: "39f40e16a389406b909677bdc1975dc9" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 39547 } }
I20260620 18:47:34.095691 28994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39f40e16a389406b909677bdc1975dc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517e56825ff4444ea601c998fd470ac1" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 43813 } } peers { permanent_uuid: "40a4841f366c47de98e1a4b354eec5ff" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 39353 } } peers { permanent_uuid: "39f40e16a389406b909677bdc1975dc9" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 39547 } } }
I20260620 18:47:34.096012 28994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39f40e16a389406b909677bdc1975dc9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:47:34.096977 28988 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39f40e16a389406b909677bdc1975dc9 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 18:47:34.106483 29005 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 39f40e16a389406b909677bdc1975dc9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:47:34.106662 29005 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:47:34.104522 28921 master_runner.cc:186] Error getting master registration for 127.5.225.190:43813: OK, instance_id { permanent_uuid: "517e56825ff4444ea601c998fd470ac1" instance_seqno: 1781981253011577 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:47:34.112814 28921 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 18:47:34.154635 28839 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.225.188" port: 39547 } attempt: 4
Jun 20 18:47:34 dist-test-slave-k6lg krb5kdc[28759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981252, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.190@KRBTEST.COM for kudu/127.5.225.188@KRBTEST.COM
I20260620 18:47:34.171592 28845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:34.155420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:39547 (local address 127.0.0.1:47146)
0620 18:47:34.155608 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:34.155626 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:34.155746 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:34.156095 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:34.156100 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:34.156634 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:47:34.156834 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:34.156848 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:34.158370 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:34.158377 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:34.159784 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:34.159799 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:34.160827 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:34.160836 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:34.161077 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:34.161817 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:47:34.161840 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:47:34.164531 (+ 2691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:47:34.168610 (+ 4079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:34.168633 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:34.168640 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:34.169204 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:34.169603 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:34.169610 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:34.169614 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:34.169787 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:34.170539 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:47:34.170544 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:47:34.170733 (+ 189us) client_negotiation.cc:769] Sending connection context
0620 18:47:34.170880 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 18:47:34.171179 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 18:47:34.171612 28991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:34.155516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47146 (local address 127.5.225.188:39547)
0620 18:47:34.155729 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:34.155735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:34.155761 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:47:34.155836 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:34.155842 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:34.155929 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:34.156047 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:34.157011 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:34.158221 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:34.159935 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:34.160670 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:34.160800 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:34.164737 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:34.164744 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:34.164749 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:34.164788 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:34.168458 (+ 3670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:34.169334 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:34.169339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:34.169344 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:34.169488 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:34.169895 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:34.169900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:34.169905 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:34.170424 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:34.170606 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:34.170998 (+ 392us) server_negotiation.cc:299] Negotiation successful
0620 18:47:34.171239 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 18:47:34.177525 28839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 517e56825ff4444ea601c998fd470ac1: Bootstrap starting.
I20260620 18:47:34.181697 28839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 517e56825ff4444ea601c998fd470ac1: Neither blocks nor log segments found. Creating new log.
I20260620 18:47:34.183028 28839 log.cc:824] T 00000000000000000000000000000000 P 517e56825ff4444ea601c998fd470ac1: Log is configured to *not* fsync() on all Append() calls
I20260620 18:47:34.185956 28839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 517e56825ff4444ea601c998fd470ac1: No bootstrap required, opened a new log
I20260620 18:47:34.193681 28839 raft_consensus.cc:348] T 00000000000000000000000000000000 P 517e56825ff4444ea601c998fd470ac1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517e56825ff4444ea601c998fd470ac1" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 43813 } } peers { permanent_uuid: "40a4841f366c47de98e1a4b354eec5ff" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 39353 } } peers { permanent_uuid: "39f40e16a389406b909677bdc1975dc9" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 39547 } }
I20260620 18:47:34.194101 28839 raft_consensus.cc:729] T 00000000000000000000000000000000 P 517e56825ff4444ea601c998fd470ac1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 517e56825ff4444ea601c998fd470ac1, State: Initialized, Role: FOLLOWER
I20260620 18:47:34.194585 28839 consensus_queue.cc:260] T 00000000000000000000000000000000 P 517e56825ff4444ea601c998fd470ac1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517e56825ff4444ea601c998fd470ac1" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 43813 } } peers { permanent_uuid: "40a4841f366c47de98e1a4b354eec5ff" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 39353 } } peers { permanent_uuid: "39f40e16a389406b909677bdc1975dc9" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 39547 } }
I20260620 18:47:34.195511 29008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 517e56825ff4444ea601c998fd470ac1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517e56825ff4444ea601c998fd470ac1" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 43813 } } peers { permanent_uuid: "40a4841f366c47de98e1a4b354eec5ff" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 39353 } } peers { permanent_uuid: "39f40e16a389406b909677bdc1975dc9" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 39547 } } }
I20260620 18:47:34.195924 29008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 517e56825ff4444ea601c998fd470ac1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:47:34.196808 28839 sys_catalog.cc:564] T 00000000000000000000000000000000 P 517e56825ff4444ea601c998fd470ac1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:47:34.196863 28814 raft_consensus.cc:482] T 00000000000000000000000000000000 P 517e56825ff4444ea601c998fd470ac1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 18:47:34.197043 28814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 517e56825ff4444ea601c998fd470ac1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:47:34.201464 28814 raft_consensus.cc:504] T 00000000000000000000000000000000 P 517e56825ff4444ea601c998fd470ac1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517e56825ff4444ea601c998fd470ac1" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 43813 } } peers { permanent_uuid: "40a4841f366c47de98e1a4b354eec5ff" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 39353 } } peers { permanent_uuid: "39f40e16a389406b909677bdc1975dc9" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 39547 } }
I20260620 18:47:34.204267 28814 leader_election.cc:288] T 00000000000000000000000000000000 P 517e56825ff4444ea601c998fd470ac1 [CANDIDATE]: Term 1 election: Requested vote from peers 40a4841f366c47de98e1a4b354eec5ff (127.5.225.189:39353), 39f40e16a389406b909677bdc1975dc9 (127.5.225.188:39547)
I20260620 18:47:34.204628 28963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "517e56825ff4444ea601c998fd470ac1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "39f40e16a389406b909677bdc1975dc9"
I20260620 18:47:34.205183 28963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39f40e16a389406b909677bdc1975dc9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:47:34.208715 28963 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 39f40e16a389406b909677bdc1975dc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 517e56825ff4444ea601c998fd470ac1 in term 1.
I20260620 18:47:34.209579 28789 leader_election.cc:302] T 00000000000000000000000000000000 P 517e56825ff4444ea601c998fd470ac1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f40e16a389406b909677bdc1975dc9, 517e56825ff4444ea601c998fd470ac1; no voters:
I20260620 18:47:34.210086 29008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 517e56825ff4444ea601c998fd470ac1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:47:34.231341 29008 raft_consensus.cc:686] T 00000000000000000000000000000000 P 517e56825ff4444ea601c998fd470ac1 [term 1 LEADER]: Becoming Leader. State: Replica: 517e56825ff4444ea601c998fd470ac1, State: Running, Role: LEADER
I20260620 18:47:34.233280 29008 consensus_queue.cc:237] T 00000000000000000000000000000000 P 517e56825ff4444ea601c998fd470ac1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517e56825ff4444ea601c998fd470ac1" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 43813 } } peers { permanent_uuid: "40a4841f366c47de98e1a4b354eec5ff" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 39353 } } peers { permanent_uuid: "39f40e16a389406b909677bdc1975dc9" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 39547 } }
I20260620 18:47:34.240680 28914 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.225.188" port: 39547 } attempt: 5
W20260620 18:47:34.241153 28942 tablet.cc:2369] T 00000000000000000000000000000000 P 39f40e16a389406b909677bdc1975dc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:47:34.248247 29015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 517e56825ff4444ea601c998fd470ac1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 517e56825ff4444ea601c998fd470ac1. Latest consensus state: current_term: 1 leader_uuid: "517e56825ff4444ea601c998fd470ac1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517e56825ff4444ea601c998fd470ac1" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 43813 } } peers { permanent_uuid: "40a4841f366c47de98e1a4b354eec5ff" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 39353 } } peers { permanent_uuid: "39f40e16a389406b909677bdc1975dc9" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 39547 } } }
I20260620 18:47:34.248698 29015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 517e56825ff4444ea601c998fd470ac1 [sys.catalog]: This master's current role is: LEADER
Jun 20 18:47:34 dist-test-slave-k6lg krb5kdc[28759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981253, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.225.189@KRBTEST.COM for kudu/127.5.225.188@KRBTEST.COM
I20260620 18:47:34.261361 28917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:34.241242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.225.188:39547 (local address 127.0.0.1:47152)
0620 18:47:34.241625 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:34.241647 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:34.241780 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:34.242780 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:34.242788 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:34.243413 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:47:34.243665 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:34.243678 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:34.245369 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:34.245373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:34.247118 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:34.247135 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:34.248349 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:34.248355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:34.248609 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:34.249326 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:47:34.249350 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:47:34.252614 (+ 3264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:47:34.257367 (+ 4753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:34.257373 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:34.257379 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:34.257989 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:34.258706 (+ 717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:34.258710 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:34.258713 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:34.258910 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:34.260186 (+ 1276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:47:34.260194 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:47:34.260371 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 18:47:34.260538 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 18:47:34.260873 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249}
I20260620 18:47:34.262775 28991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:34.242026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47152 (local address 127.5.225.188:39547)
0620 18:47:34.242220 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:34.242225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:34.242244 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:47:34.242311 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:34.242319 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:34.242442 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:34.242556 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:34.243861 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:34.245201 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:34.247316 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:34.247989 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:34.248127 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:34.252926 (+ 4799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:34.252933 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:34.252938 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:34.252978 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:34.257199 (+ 4221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:34.258198 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:34.258205 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:34.258211 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:34.258385 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:34.259107 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:34.259112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:34.259115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:34.259490 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:34.261636 (+ 2146us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:34.261760 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 18:47:34.262023 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 18:47:34.268915 28793 tablet.cc:2369] T 00000000000000000000000000000000 P 517e56825ff4444ea601c998fd470ac1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:47:34.268999 28914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40a4841f366c47de98e1a4b354eec5ff: Bootstrap starting.
I20260620 18:47:34.273900 28914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40a4841f366c47de98e1a4b354eec5ff: Neither blocks nor log segments found. Creating new log.
I20260620 18:47:34.275637 28914 log.cc:824] T 00000000000000000000000000000000 P 40a4841f366c47de98e1a4b354eec5ff: Log is configured to *not* fsync() on all Append() calls
I20260620 18:47:34.276948 29027 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:47:34.284862 29027 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:47:34.291646 28918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:34.236402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39864 (local address 127.5.225.189:39353)
0620 18:47:34.236645 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:34.236655 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:34.254524 (+ 17869us) server_negotiation.cc:407] Connection header received
0620 18:47:34.254846 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:34.254855 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:34.254955 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:34.255037 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:34.258195 (+ 3158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:34.259697 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:34.264261 (+ 4564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:34.264960 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:34.265093 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:34.271141 (+ 6048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:34.271151 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:34.271156 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:34.271195 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:34.274968 (+ 3773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:34.284562 (+ 9594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:34.284569 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:34.284581 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:34.284759 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:34.286638 (+ 1879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:34.286644 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:34.286648 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:34.287034 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:34.287283 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:34.290954 (+ 3671us) server_negotiation.cc:299] Negotiation successful
0620 18:47:34.291244 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 18:47:34.295603 28914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40a4841f366c47de98e1a4b354eec5ff: No bootstrap required, opened a new log
I20260620 18:47:34.302559 28842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:34.238478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37704 (local address 127.5.225.190:43813)
0620 18:47:34.238731 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:34.238739 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:34.254893 (+ 16154us) server_negotiation.cc:407] Connection header received
0620 18:47:34.255622 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:34.255628 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:34.255721 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:34.255804 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:34.269830 (+ 14026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:34.270944 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:34.277928 (+ 6984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:34.278773 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:34.278912 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:34.295023 (+ 16111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:34.295031 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:34.295036 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:34.295093 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:34.298956 (+ 3863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:34.299853 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:34.299861 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:34.299866 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:34.300082 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:34.300572 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:34.300577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:34.300580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:34.300975 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:34.301242 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:34.301849 (+ 607us) server_negotiation.cc:299] Negotiation successful
0620 18:47:34.302106 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:47:34.311544 29025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:34.253952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47164 (local address 127.5.225.188:39547)
0620 18:47:34.262752 (+ 8800us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:34.262765 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:34.263620 (+ 855us) server_negotiation.cc:407] Connection header received
0620 18:47:34.263668 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:34.263685 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:34.263756 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:34.263851 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:34.266149 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:34.268199 (+ 2050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:34.287606 (+ 19407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:34.288749 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:34.288944 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:34.299552 (+ 10608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:34.299616 (+ 64us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:34.299625 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:34.299698 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:34.305843 (+ 6145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:34.307203 (+ 1360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:34.307208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:34.307212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:34.307484 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:34.308247 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:34.308253 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:34.308284 (+ 31us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:34.308901 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:34.309345 (+ 444us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:34.310330 (+ 985us) server_negotiation.cc:299] Negotiation successful
0620 18:47:34.310776 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8581,"thread_start_us":8380,"threads_started":1}
I20260620 18:47:34.323378 28963 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 39f40e16a389406b909677bdc1975dc9 [term 1 FOLLOWER]: Refusing update from remote peer 517e56825ff4444ea601c998fd470ac1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:47:34.325846 28914 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40a4841f366c47de98e1a4b354eec5ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517e56825ff4444ea601c998fd470ac1" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 43813 } } peers { permanent_uuid: "40a4841f366c47de98e1a4b354eec5ff" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 39353 } } peers { permanent_uuid: "39f40e16a389406b909677bdc1975dc9" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 39547 } }
I20260620 18:47:34.326534 28914 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40a4841f366c47de98e1a4b354eec5ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40a4841f366c47de98e1a4b354eec5ff, State: Initialized, Role: FOLLOWER
I20260620 18:47:34.327188 29015 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 517e56825ff4444ea601c998fd470ac1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f40e16a389406b909677bdc1975dc9" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 39547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:47:34.327510 28914 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40a4841f366c47de98e1a4b354eec5ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517e56825ff4444ea601c998fd470ac1" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 43813 } } peers { permanent_uuid: "40a4841f366c47de98e1a4b354eec5ff" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 39353 } } peers { permanent_uuid: "39f40e16a389406b909677bdc1975dc9" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 39547 } }
I20260620 18:47:34.344883 28994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39f40e16a389406b909677bdc1975dc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 517e56825ff4444ea601c998fd470ac1. Latest consensus state: current_term: 1 leader_uuid: "517e56825ff4444ea601c998fd470ac1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517e56825ff4444ea601c998fd470ac1" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 43813 } } peers { permanent_uuid: "40a4841f366c47de98e1a4b354eec5ff" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 39353 } } peers { permanent_uuid: "39f40e16a389406b909677bdc1975dc9" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 39547 } } }
I20260620 18:47:34.345244 28994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39f40e16a389406b909677bdc1975dc9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:47:34.347513 29035 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 18:47:34.349002 29033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40a4841f366c47de98e1a4b354eec5ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517e56825ff4444ea601c998fd470ac1" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 43813 } } peers { permanent_uuid: "40a4841f366c47de98e1a4b354eec5ff" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 39353 } } peers { permanent_uuid: "39f40e16a389406b909677bdc1975dc9" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 39547 } } }
I20260620 18:47:34.349481 29033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40a4841f366c47de98e1a4b354eec5ff [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:47:34.350440 29015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 517e56825ff4444ea601c998fd470ac1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "517e56825ff4444ea601c998fd470ac1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517e56825ff4444ea601c998fd470ac1" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 43813 } } peers { permanent_uuid: "40a4841f366c47de98e1a4b354eec5ff" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 39353 } } peers { permanent_uuid: "39f40e16a389406b909677bdc1975dc9" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 39547 } } }
I20260620 18:47:34.350715 29015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 517e56825ff4444ea601c998fd470ac1 [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:34.351109 29032 mvcc.cc:205] Tried to move back new op lower bound from 7298995217698160640 to 7298995217346113536. Current Snapshot: MvccSnapshot[applied={T|T < 7298995217698160640}]
I20260620 18:47:34.351398 29034 mvcc.cc:205] Tried to move back new op lower bound from 7298995217698160640 to 7298995217346113536. Current Snapshot: MvccSnapshot[applied={T|T < 7298995217698160640}]
I20260620 18:47:34.352067 28994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39f40e16a389406b909677bdc1975dc9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "517e56825ff4444ea601c998fd470ac1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517e56825ff4444ea601c998fd470ac1" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 43813 } } peers { permanent_uuid: "40a4841f366c47de98e1a4b354eec5ff" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 39353 } } peers { permanent_uuid: "39f40e16a389406b909677bdc1975dc9" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 39547 } } }
I20260620 18:47:34.352303 28994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39f40e16a389406b909677bdc1975dc9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:47:34.353087 28914 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40a4841f366c47de98e1a4b354eec5ff [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:47:34.353150 28889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "517e56825ff4444ea601c998fd470ac1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "40a4841f366c47de98e1a4b354eec5ff"
I20260620 18:47:34.353609 28889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40a4841f366c47de98e1a4b354eec5ff [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:47:34.354317 29027 catalog_manager.cc:1347] Generated new cluster ID: 8cea57b68fc74bf391e41a0190324bc1
I20260620 18:47:34.354400 29027 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:47:34.358429 28889 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 40a4841f366c47de98e1a4b354eec5ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 517e56825ff4444ea601c998fd470ac1 in term 1.
I20260620 18:47:34.359479 28888 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 40a4841f366c47de98e1a4b354eec5ff [term 1 FOLLOWER]: Refusing update from remote peer 517e56825ff4444ea601c998fd470ac1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:47:34.362053 29008 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 517e56825ff4444ea601c998fd470ac1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40a4841f366c47de98e1a4b354eec5ff" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:47:34.360684 28772 master_runner.cc:186] Error getting master registration for 127.5.225.189:39353: OK, instance_id { permanent_uuid: "40a4841f366c47de98e1a4b354eec5ff" instance_seqno: 1781981253509945 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:47:34.372617 28772 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 18:47:34.407878 29027 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:47:34.411590 29027 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:47:34.427055 29033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40a4841f366c47de98e1a4b354eec5ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 517e56825ff4444ea601c998fd470ac1. Latest consensus state: current_term: 1 leader_uuid: "517e56825ff4444ea601c998fd470ac1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517e56825ff4444ea601c998fd470ac1" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 43813 } } peers { permanent_uuid: "40a4841f366c47de98e1a4b354eec5ff" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 39353 } } peers { permanent_uuid: "39f40e16a389406b909677bdc1975dc9" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 39547 } } }
I20260620 18:47:34.427402 29033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40a4841f366c47de98e1a4b354eec5ff [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:47:34.432098 29008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 517e56825ff4444ea601c998fd470ac1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "517e56825ff4444ea601c998fd470ac1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517e56825ff4444ea601c998fd470ac1" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 43813 } } peers { permanent_uuid: "40a4841f366c47de98e1a4b354eec5ff" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 39353 } } peers { permanent_uuid: "39f40e16a389406b909677bdc1975dc9" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 39547 } } }
I20260620 18:47:34.432407 29008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 517e56825ff4444ea601c998fd470ac1 [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:34.433275 29049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40a4841f366c47de98e1a4b354eec5ff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "517e56825ff4444ea601c998fd470ac1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517e56825ff4444ea601c998fd470ac1" member_type: VOTER last_known_addr { host: "127.5.225.190" port: 43813 } } peers { permanent_uuid: "40a4841f366c47de98e1a4b354eec5ff" member_type: VOTER last_known_addr { host: "127.5.225.189" port: 39353 } } peers { permanent_uuid: "39f40e16a389406b909677bdc1975dc9" member_type: VOTER last_known_addr { host: "127.5.225.188" port: 39547 } } }
I20260620 18:47:34.433768 29049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40a4841f366c47de98e1a4b354eec5ff [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:47:34.441675 29053 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 18:47:34.449855 29027 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 517e56825ff4444ea601c998fd470ac1: Generated new TSK 0
I20260620 18:47:34.450760 29027 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 18:47:34.558645 28842 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:47:34.546674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37706 (local address 127.5.225.190:43813)
0620 18:47:34.546888 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:34.546894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:34.555063 (+ 8169us) server_negotiation.cc:407] Connection header received
0620 18:47:34.556738 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:34.556774 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:34.556887 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:34.556982 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:34.558354 (+ 1372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37706: BlockingRecv error: recv got EOF from 127.0.0.1:37706 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 18:47:34.561200 28918 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:47:34.546365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39866 (local address 127.5.225.189:39353)
0620 18:47:34.546594 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:34.546601 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:34.559292 (+ 12691us) server_negotiation.cc:407] Connection header received
0620 18:47:34.559548 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:34.559553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:34.559636 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:34.559709 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:34.560969 (+ 1260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39866: BlockingRecv error: recv got EOF from 127.0.0.1:39866 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 18:47:34.562428 29025 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:47:34.555681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47178 (local address 127.5.225.188:39547)
0620 18:47:34.555901 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:34.555909 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:34.560167 (+ 4258us) server_negotiation.cc:407] Connection header received
0620 18:47:34.560978 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:34.560984 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:34.561081 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:34.561167 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:34.562140 (+ 973us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47178: BlockingRecv error: recv got EOF from 127.0.0.1:47178 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 18:47:34.569368 28918 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:47:34.567192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39882 (local address 127.5.225.189:39353)
0620 18:47:34.567370 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:34.567375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:34.567876 (+ 501us) server_negotiation.cc:407] Connection header received
0620 18:47:34.568098 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:34.568103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:34.568236 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:34.568341 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:34.569105 (+ 764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39882: BlockingRecv error: recv got EOF from 127.0.0.1:39882 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 18:47:34.575824 28918 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:47:34.573821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39892 (local address 127.5.225.189:39353)
0620 18:47:34.573984 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:34.573990 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:34.574482 (+ 492us) server_negotiation.cc:407] Connection header received
0620 18:47:34.574680 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:34.574686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:34.574768 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:34.574869 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:34.575621 (+ 752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39892: BlockingRecv error: recv got EOF from 127.0.0.1:39892 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260620 18:47:34.583252 28918 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:47:34.581082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39900 (local address 127.5.225.189:39353)
0620 18:47:34.581236 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:34.581241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:34.581712 (+ 471us) server_negotiation.cc:407] Connection header received
0620 18:47:34.581883 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:34.581889 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:34.581985 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:34.582103 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:34.583030 (+ 927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39900: BlockingRecv error: recv got EOF from 127.0.0.1:39900 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 18:47:34.592914 28918 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:47:34.590817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39906 (local address 127.5.225.189:39353)
0620 18:47:34.591016 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:34.591022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:34.591519 (+ 497us) server_negotiation.cc:407] Connection header received
0620 18:47:34.591728 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:34.591733 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:34.591817 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:34.591918 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:34.592708 (+ 790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39906: BlockingRecv error: recv got EOF from 127.0.0.1:39906 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
W20260620 18:47:34.606308 28918 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:47:34.604201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39922 (local address 127.5.225.189:39353)
0620 18:47:34.604370 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:34.604376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:34.604872 (+ 496us) server_negotiation.cc:407] Connection header received
0620 18:47:34.605076 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:34.605083 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:34.605167 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:34.605269 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:34.606054 (+ 785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39922: BlockingRecv error: recv got EOF from 127.0.0.1:39922 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 18:47:34.628163 28918 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:47:34.625871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39926 (local address 127.5.225.189:39353)
0620 18:47:34.626061 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:34.626067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:34.626657 (+ 590us) server_negotiation.cc:407] Connection header received
0620 18:47:34.626857 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:34.626862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:34.626953 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:34.627059 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:34.627942 (+ 883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39926: BlockingRecv error: recv got EOF from 127.0.0.1:39926 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 18:47:34.670117 28918 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:47:34.667764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39934 (local address 127.5.225.189:39353)
0620 18:47:34.667947 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:34.667953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:34.668517 (+ 564us) server_negotiation.cc:407] Connection header received
0620 18:47:34.668719 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:34.668725 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:34.668841 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:34.668963 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:34.669823 (+ 860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39934: BlockingRecv error: recv got EOF from 127.0.0.1:39934 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 18:47:34.739705 28918 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:47:34.737655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39940 (local address 127.5.225.189:39353)
0620 18:47:34.737889 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:34.737895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:34.738259 (+ 364us) server_negotiation.cc:407] Connection header received
0620 18:47:34.738433 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:34.738438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:34.738541 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:34.738681 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:34.739482 (+ 801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39940: BlockingRecv error: recv got EOF from 127.0.0.1:39940 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260620 18:47:34.873642 28918 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:47:34.871818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39948 (local address 127.5.225.189:39353)
0620 18:47:34.871990 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:34.871995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:34.872335 (+ 340us) server_negotiation.cc:407] Connection header received
0620 18:47:34.872527 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:34.872533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:34.872622 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:34.872762 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:34.873456 (+ 694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39948: BlockingRecv error: recv got EOF from 127.0.0.1:39948 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:47:35.108820 29005 catalog_manager.cc:1259] Loaded cluster ID: 8cea57b68fc74bf391e41a0190324bc1
I20260620 18:47:35.108930 29005 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 39f40e16a389406b909677bdc1975dc9: loading cluster ID for follower catalog manager: success
I20260620 18:47:35.111979 29005 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 39f40e16a389406b909677bdc1975dc9: acquiring CA information for follower catalog manager: success
I20260620 18:47:35.113886 29005 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 39f40e16a389406b909677bdc1975dc9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 18:47:35.137492 28918 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:47:35.135905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39950 (local address 127.5.225.189:39353)
0620 18:47:35.136080 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:35.136085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:35.136330 (+ 245us) server_negotiation.cc:407] Connection header received
0620 18:47:35.136499 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:35.136502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:35.136572 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:35.136667 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:35.137334 (+ 667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39950: BlockingRecv error: recv got EOF from 127.0.0.1:39950 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:47:35.446269 29053 catalog_manager.cc:1259] Loaded cluster ID: 8cea57b68fc74bf391e41a0190324bc1
I20260620 18:47:35.446422 29053 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 40a4841f366c47de98e1a4b354eec5ff: loading cluster ID for follower catalog manager: success
I20260620 18:47:35.448942 29053 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 40a4841f366c47de98e1a4b354eec5ff: acquiring CA information for follower catalog manager: success
I20260620 18:47:35.450949 29053 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 40a4841f366c47de98e1a4b354eec5ff: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:47:35.662194 29122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:35.655287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39956 (local address 127.5.225.189:39353)
0620 18:47:35.656086 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:35.656091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:35.656108 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:47:35.656174 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:35.656178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:35.656305 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:35.656412 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:47:35.656975 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:35.657794 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:35.659842 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:35.660283 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:35.660383 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:35.660967 (+ 584us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:47:35.661485 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:47:35.661594 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:35.661742 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 18:47:35.661806 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":539,"threads_started":1}
I20260620 18:47:35.675192 29128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:35.668434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47190 (local address 127.5.225.188:39547)
0620 18:47:35.668950 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:35.668956 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:35.669251 (+ 295us) server_negotiation.cc:407] Connection header received
0620 18:47:35.669451 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:35.669455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:35.669590 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:35.669727 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:47:35.670260 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:35.670922 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:35.672881 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:35.673349 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:35.673447 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:35.674005 (+ 558us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:47:35.674553 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:47:35.674667 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:35.674793 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 18:47:35.674859 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":251,"threads_started":1}
I20260620 18:47:35.700208 29122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:35.681338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39962 (local address 127.5.225.189:39353)
0620 18:47:35.681541 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:35.681547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:35.692492 (+ 10945us) server_negotiation.cc:407] Connection header received
0620 18:47:35.692763 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:35.692768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:35.692850 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:35.692924 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:47:35.693758 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:35.694986 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:35.697796 (+ 2810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:35.698466 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:35.698566 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:35.699307 (+ 741us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:47:35.699447 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:47:35.699519 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:35.699798 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 18:47:35.699886 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 18:47:35.700995 29135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:35.681573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37712 (local address 127.5.225.190:43813)
0620 18:47:35.691521 (+ 9948us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:35.691530 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:35.692469 (+ 939us) server_negotiation.cc:407] Connection header received
0620 18:47:35.692604 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:35.692609 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:35.692887 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:35.693044 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:47:35.693829 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:35.694925 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:35.697427 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:35.698171 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:35.698325 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:35.699208 (+ 883us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:47:35.699749 (+ 541us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:47:35.699840 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:35.700630 (+ 790us) server_negotiation.cc:299] Negotiation successful
0620 18:47:35.700714 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9775,"thread_start_us":9955,"threads_started":1}
I20260620 18:47:35.702843 29128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:35.692725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47192 (local address 127.5.225.188:39547)
0620 18:47:35.692892 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:35.692900 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:35.696118 (+ 3218us) server_negotiation.cc:407] Connection header received
0620 18:47:35.696296 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:35.696301 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:35.696397 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:35.696498 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:47:35.697409 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:35.698559 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:35.700865 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:35.701464 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:35.701596 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:35.702086 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:47:35.702258 (+ 172us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:47:35.702311 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:35.702507 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 18:47:35.702582 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:47:35.706908 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 28772
I20260620 18:47:35.707926 28834 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:47:35.820915 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 28772
I20260620 18:47:35.841727 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 28847
I20260620 18:47:35.842957 28909 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:47:35.949301 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 28847
I20260620 18:47:35.969394 6022 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuQUw40/build/asan/bin/kudu pid 28921
I20260620 18:47:35.970613 28983 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:47:36.078451 6022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuQUw40/build/asan/bin/kudu with pid 28921
2026-06-20T18:47:36Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4874 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10066 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (361572 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST